[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:10 dist-test-slave-44kr krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:10 dist-test-slave-44kr krb5kdc[399](info): set up 2 sockets
Sep 27 02:42:10 dist-test-slave-44kr krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 02:42:11.052623 392 mini_kdc.cc:121] Time spent starting KDC: real 1.781s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:11.177210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41535
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:11.261591 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:11.261842 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:11.261866 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:11.263177 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:11.263242 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:11.263255 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:11.263267 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:11.263278 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:11.264966 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:11.265218 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:11.265481 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:11.271497 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:11.271559 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:11.271780 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:11.271656 412 server_base.cc:1034] running on GCE node
I20260927 02:42:11.272012 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:11.272310 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:11.273452 412 hybrid_clock.cc:648] HybridClock initialized: now 1790476931273437 us; error 39 us; skew 500 ppm
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:11.276059 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:11.276551 412 webserver.cc:460] Webserver started at http://127.0.98.62:44003/ using document root <none> and password file <none>
I20260927 02:42:11.276793 412 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:11.276841 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:11.276962 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:11.278059 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "fa20900df986e70ebf445a58d94ac3d7"
server_key_iv: "6be786252e854f891332ac960a49289d"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.278407 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "fa20900df986e70ebf445a58d94ac3d7"
server_key_iv: "6be786252e854f891332ac960a49289d"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.279807 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:11.280563 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.280766 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:11.280853 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "fa20900df986e70ebf445a58d94ac3d7"
server_key_iv: "6be786252e854f891332ac960a49289d"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.280915 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:11.287966 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:11.289983 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:11.290124 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:11.293833 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41535
I20260927 02:42:11.293866 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41535 every 8 connection(s)
I20260927 02:42:11.294195 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:11.295061 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:11.297546 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49: Bootstrap starting.
I20260927 02:42:11.298130 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:11.298395 480 log.cc:824] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:11.299105 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49: No bootstrap required, opened a new log
I20260927 02:42:11.300841 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41535 } }
I20260927 02:42:11.300998 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:11.301038 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e87ff23c37f454a9aa3b2bc5c9b1d49, State: Initialized, Role: FOLLOWER
I20260927 02:42:11.301172 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41535 } }
I20260927 02:42:11.301255 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:11.301278 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:11.301309 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:11.301914 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41535 } }
I20260927 02:42:11.302003 480 leader_election.cc:302] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e87ff23c37f454a9aa3b2bc5c9b1d49; no voters:
I20260927 02:42:11.302198 480 leader_election.cc:288] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:11.302284 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:11.302387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 412
I20260927 02:42:11.302448 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [term 1 LEADER]: Becoming Leader. State: Replica: 7e87ff23c37f454a9aa3b2bc5c9b1d49, State: Running, Role: LEADER
I20260927 02:42:11.302470 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:11.302488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:11.302548 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41535 } }
I20260927 02:42:11.303032 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e87ff23c37f454a9aa3b2bc5c9b1d49. Latest consensus state: current_term: 1 leader_uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41535 } } }
I20260927 02:42:11.303114 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:11.303278 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e87ff23c37f454a9aa3b2bc5c9b1d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41535 } } }
I20260927 02:42:11.303305 392 external_mini_cluster.cc:1428] Setting key d00aba27d3accd24956e7072f360e9fd
I20260927 02:42:11.303336 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:11.303565 488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:11.304423 488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:11.307489 488 catalog_manager.cc:1347] Generated new cluster ID: 50c69fab15e4474f803e0946040ca150
I20260927 02:42:11.307555 488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:11.321745 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:11.304275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.0.98.62:41535)
0927 02:42:11.304595 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:11.304606 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:11.304876 (+ 270us) server_negotiation.cc:407] Connection header received
0927 02:42:11.307145 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:11.307151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:11.307193 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:11.307345 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:11.309452 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.310898 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.312445 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.313091 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.313214 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:11.317647 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:11.317651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:11.317651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:11.317664 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:11.319560 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.320072 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.320075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.320076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.320188 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.320508 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.320509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.320510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.320879 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:11.321031 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:11.321353 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 02:42:11.321494 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260927 02:42:11.326834 488 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:11.327525 488 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:11.334800 488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e87ff23c37f454a9aa3b2bc5c9b1d49: Generated new TSK 0
I20260927 02:42:11.334992 488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:11.364929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41535
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:11.446725 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:11.446954 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:11.446978 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:11.448196 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:11.448246 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:11.448272 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:11.449801 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:11.450031 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:11.450270 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:11.455514 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:11.455638 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:11.455514 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:11.455686 506 server_base.cc:1034] running on GCE node
I20260927 02:42:11.455998 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:11.456279 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:11.457425 506 hybrid_clock.cc:648] HybridClock initialized: now 1790476931457410 us; error 31 us; skew 500 ppm
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:11.460076 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:11.460636 506 webserver.cc:460] Webserver started at http://127.0.98.1:36999/ using document root <none> and password file <none>
I20260927 02:42:11.460835 506 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:11.460883 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:11.460999 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:11.462088 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "83676821e91b43338ef69f006850f3cb"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "c3b9404dd50a0cbbe5e9a929d3256e9e"
server_key_iv: "982636eddeeace82729172551050c1db"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.462469 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83676821e91b43338ef69f006850f3cb"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "c3b9404dd50a0cbbe5e9a929d3256e9e"
server_key_iv: "982636eddeeace82729172551050c1db"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.463748 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:11.464468 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.464620 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:11.464700 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "83676821e91b43338ef69f006850f3cb"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "c3b9404dd50a0cbbe5e9a929d3256e9e"
server_key_iv: "982636eddeeace82729172551050c1db"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.464769 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:11.485244 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:11.487080 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:11.487211 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:11.487452 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:11.487764 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:11.487799 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.487828 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:11.487845 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.493491 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36771
I20260927 02:42:11.493636 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36771 every 8 connection(s)
I20260927 02:42:11.493870 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:11.499804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 506
I20260927 02:42:11.499904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:11.500058 392 external_mini_cluster.cc:1428] Setting key e9936a67ff202691cfc38303f90f44b4
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:11.503578 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:11.494605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56811 (local address 127.0.98.62:41535)
0927 02:42:11.494674 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:11.494675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:11.494899 (+ 224us) server_negotiation.cc:407] Connection header received
0927 02:42:11.495370 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:11.495371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:11.495381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:11.495432 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:11.496365 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.496973 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.498049 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.498253 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.498307 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:11.500683 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:11.500685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:11.500685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:11.500693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:11.502172 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.502607 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.502608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.502608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.502670 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.502893 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.502895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.502895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.503012 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:11.503069 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:11.503388 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 02:42:11.503481 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:42:11.503702 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:11.494619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41535 (local address 127.0.98.1:56811)
0927 02:42:11.494781 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:11.494818 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:11.495292 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:11.495488 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:11.495492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:11.496044 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:11.496256 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:11.496261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.497066 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.497067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:11.497965 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:11.497967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.498336 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.498337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:11.498443 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:11.498931 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:11.498939 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:11.500597 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:11.502251 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:11.502257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:11.502259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:11.502514 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:11.502731 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:11.502732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:11.502732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:11.502832 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:11.503066 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:11.503073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:11.503203 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:42:11.503356 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 02:42:11.503509 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260927 02:42:11.504177 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41535
I20260927 02:42:11.504266 635 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:11.504483 635 heartbeater.cc:511] Master 127.0.98.62:41535 requested a full tablet report, sending...
I20260927 02:42:11.505009 444 ts_manager.cc:194] Registered new tserver with Master: 83676821e91b43338ef69f006850f3cb (127.0.98.1:36771)
I20260927 02:42:11.505620 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56811
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:11.527829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41535
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:11.610383 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:11.610607 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:11.610642 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:11.611855 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:11.611905 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:11.611943 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:11.613541 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:11.613813 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:11.614034 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:11.619134 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:11.619119 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:11.619282 640 server_base.cc:1034] running on GCE node
W20260927 02:42:11.619119 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:11.619633 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:11.619865 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:11.621027 640 hybrid_clock.cc:648] HybridClock initialized: now 1790476931621010 us; error 33 us; skew 500 ppm
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:11.623584 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:11.624042 640 webserver.cc:460] Webserver started at http://127.0.98.2:34287/ using document root <none> and password file <none>
I20260927 02:42:11.624295 640 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:11.624346 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:11.624495 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:11.625527 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "af1c9463038f42a188ae9cfdf60373d6"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "ced379c155ca0981d7338607d16cb762"
server_key_iv: "cd401ec51e3a0e642cb46a53658e7aff"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.625850 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af1c9463038f42a188ae9cfdf60373d6"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "ced379c155ca0981d7338607d16cb762"
server_key_iv: "cd401ec51e3a0e642cb46a53658e7aff"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.627170 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:42:11.627941 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.628321 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:11.628441 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "af1c9463038f42a188ae9cfdf60373d6"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "ced379c155ca0981d7338607d16cb762"
server_key_iv: "cd401ec51e3a0e642cb46a53658e7aff"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.628506 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:11.655223 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:11.657660 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:11.657823 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:11.658063 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:11.658376 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:11.658412 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.658442 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:11.658459 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.664624 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43799
I20260927 02:42:11.664671 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43799 every 8 connection(s)
I20260927 02:42:11.664981 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:11.673070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 640
I20260927 02:42:11.673162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:11.673341 392 external_mini_cluster.cc:1428] Setting key e4f953eb7fe023abfd19ac2dfb469d48
I20260927 02:42:11.674217 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:11.665662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41865 (local address 127.0.98.62:41535)
0927 02:42:11.665732 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:11.665734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:11.665880 (+ 146us) server_negotiation.cc:407] Connection header received
0927 02:42:11.666404 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:11.666405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:11.666417 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:11.666469 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:11.667519 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.668007 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.669111 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.669313 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.669375 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:11.671317 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:11.671318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:11.671318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:11.671324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:11.672586 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.673065 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.673067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.673067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.673127 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.673443 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.673445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.673445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.673669 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:11.673730 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:11.674041 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 02:42:11.674126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:42:11.674429 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:11.665623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41535 (local address 127.0.98.2:41865)
0927 02:42:11.665756 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:11.665794 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:11.666304 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:11.666542 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:11.666546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:11.667168 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:11.667371 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:11.667376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.668089 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.668090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:11.669026 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:11.669030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.669383 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.669384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:11.669483 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:11.669925 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:11.669932 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:11.671231 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:11.672672 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:11.672680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:11.672682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:11.672968 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:11.673206 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:11.673206 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:11.673207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:11.673351 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:11.673749 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:11.673756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:11.673903 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 02:42:11.674013 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 02:42:11.674198 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260927 02:42:11.675015 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41535
I20260927 02:42:11.675169 769 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:11.675377 769 heartbeater.cc:511] Master 127.0.98.62:41535 requested a full tablet report, sending...
I20260927 02:42:11.675740 444 ts_manager.cc:194] Registered new tserver with Master: af1c9463038f42a188ae9cfdf60373d6 (127.0.98.2:43799)
I20260927 02:42:11.676110 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41865
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:11.699673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41535
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:11.784703 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:11.784930 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:11.784957 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:11.786199 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:11.786254 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:11.786281 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:11.787799 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:11.788038 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:11.788254 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:11.793527 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:11.793495 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:11.793488 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:11.794199 774 server_base.cc:1034] running on GCE node
I20260927 02:42:11.794411 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:11.794656 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:11.795811 774 hybrid_clock.cc:648] HybridClock initialized: now 1790476931795791 us; error 30 us; skew 500 ppm
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:11.798286 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:11.798744 774 webserver.cc:460] Webserver started at http://127.0.98.3:42531/ using document root <none> and password file <none>
I20260927 02:42:11.798978 774 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:11.799038 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:11.799175 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:11.800197 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "3da1f10bcc4e4fa189878f39baf6e30d"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "b01100055309d2a136ebf952a29d4130"
server_key_iv: "eca5dc22bc3c70e3031f64829869e109"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.800554 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3da1f10bcc4e4fa189878f39baf6e30d"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "b01100055309d2a136ebf952a29d4130"
server_key_iv: "eca5dc22bc3c70e3031f64829869e109"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.801849 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:11.802702 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.803001 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:11.803081 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "3da1f10bcc4e4fa189878f39baf6e30d"
format_stamp: "Formatted at 2026-09-27 02:42:11 on dist-test-slave-44kr"
server_key: "b01100055309d2a136ebf952a29d4130"
server_key_iv: "eca5dc22bc3c70e3031f64829869e109"
server_key_version: "encryptionkey@0"
I20260927 02:42:11.803135 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:11.822702 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:11.824824 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:11.824935 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:11.825125 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:11.825425 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:11.825459 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.825487 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:11.825511 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:11.831522 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40237
I20260927 02:42:11.831585 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40237 every 8 connection(s)
I20260927 02:42:11.831918 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:11.834913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 774
I20260927 02:42:11.834983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:11.835115 392 external_mini_cluster.cc:1428] Setting key 9a3b2a2f7923f88b1cc1d37888b76b1a
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:11.840831 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:11.832608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38435 (local address 127.0.98.62:41535)
0927 02:42:11.832676 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:11.832678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:11.832883 (+ 205us) server_negotiation.cc:407] Connection header received
0927 02:42:11.833376 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:11.833378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:11.833390 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:11.833438 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:11.834385 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.834988 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.836219 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.836451 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.836503 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:11.838444 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:11.838445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:11.838446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:11.838453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:11.839619 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.839967 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.839969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.839969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.840024 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.840232 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.840233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.840233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.840340 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:11.840414 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:11.840668 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 02:42:11.840742 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:42:11.841037 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:11.832610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41535 (local address 127.0.98.3:38435)
0927 02:42:11.832767 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:11.832802 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:11.833284 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:11.833519 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:11.833523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:11.834092 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:11.834289 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:11.834295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.835065 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.835067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:11.836150 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:11.836153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.836508 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.836509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:11.836644 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:11.837085 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:11.837093 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:11.838345 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:11.839687 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:11.839693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:11.839695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:11.839905 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:11.840075 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:11.840076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:11.840076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:11.840179 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:11.840409 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:11.840414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:11.840547 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 02:42:11.840641 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:42:11.840790 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260927 02:42:11.841464 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41535
I20260927 02:42:11.841542 903 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:11.841703 903 heartbeater.cc:511] Master 127.0.98.62:41535 requested a full tablet report, sending...
I20260927 02:42:11.842077 444 ts_manager.cc:194] Registered new tserver with Master: 3da1f10bcc4e4fa189878f39baf6e30d (127.0.98.3:40237)
I20260927 02:42:11.842430 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38435
I20260927 02:42:11.846975 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:11.870608 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:11.862317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.98.1:36771)
0927 02:42:11.862458 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:11.862464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:11.862480 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:42:11.862554 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:11.862556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:11.862571 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:11.862636 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:11.863233 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.863968 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.864918 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.865253 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.865318 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:11.867857 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:11.867865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:11.867866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:11.867873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:11.869568 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.869897 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.869900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.869900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.869956 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.870087 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.870088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.870088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.870217 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:11.870345 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:11.870461 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 02:42:11.870526 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
W20260927 02:42:11.871002 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42688
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:11.893178 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:11.886057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.0.98.1:36771)
0927 02:42:11.886111 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:11.886113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:11.886141 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:42:11.886208 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:11.886209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:11.886223 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:11.886263 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:11.886796 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.887281 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.888011 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:11.888214 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:11.888271 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:11.890636 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:11.890637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:11.890637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:11.890645 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:11.892084 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.892484 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.892485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.892486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.892544 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:11.892712 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:11.892712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:11.892713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:11.892810 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:11.892917 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:11.892998 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 02:42:11.893082 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:42:11.893844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 506
I20260927 02:42:11.898169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 640
I20260927 02:42:11.903059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 774
I20260927 02:42:11.907145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 412
2026-09-27T02:42:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2660 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[917](info): set up 2 sockets
Sep 27 02:42:11 dist-test-slave-44kr krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 02:42:12.864005 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:12 dist-test-slave-44kr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:12.983747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41059
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:13.063128 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:13.063380 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:13.063405 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:13.064716 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:13.064770 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:13.064783 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:13.064795 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:13.064806 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:13.066668 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:13.067018 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:13.067274 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:13.072490 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:13.072480 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:13.072479 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:13.072679 930 server_base.cc:1034] running on GCE node
I20260927 02:42:13.072969 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:13.073275 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:13.074445 930 hybrid_clock.cc:648] HybridClock initialized: now 1790476933074427 us; error 36 us; skew 500 ppm
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:13.077159 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:13.077584 930 webserver.cc:460] Webserver started at http://127.0.98.62:37599/ using document root <none> and password file <none>
I20260927 02:42:13.077775 930 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:13.077818 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:13.077906 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:13.078955 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "d6c2cc88ae3d431aba89b6b35cc2efce"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "9642e5246f340d45b86e4a3552e1d385"
server_key_iv: "5b735c56b906bcbb1711cce248c3157b"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.079286 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6c2cc88ae3d431aba89b6b35cc2efce"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "9642e5246f340d45b86e4a3552e1d385"
server_key_iv: "5b735c56b906bcbb1711cce248c3157b"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.080696 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:13.081548 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.081735 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:13.081818 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "d6c2cc88ae3d431aba89b6b35cc2efce"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "9642e5246f340d45b86e4a3552e1d385"
server_key_iv: "5b735c56b906bcbb1711cce248c3157b"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.081887 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:13.100539 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:13.102602 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:13.102718 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:13.106451 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41059
I20260927 02:42:13.106499 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41059 every 8 connection(s)
I20260927 02:42:13.106810 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:13.107750 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:13.108544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 930
I20260927 02:42:13.108639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:13.108780 392 external_mini_cluster.cc:1428] Setting key bc68cf0e451e276f9244601f78cbf9af
I20260927 02:42:13.110808 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce: Bootstrap starting.
I20260927 02:42:13.111359 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:13.111629 998 log.cc:824] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:13.112329 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce: No bootstrap required, opened a new log
I20260927 02:42:13.114795 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c2cc88ae3d431aba89b6b35cc2efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } }
I20260927 02:42:13.114950 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:13.114972 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c2cc88ae3d431aba89b6b35cc2efce, State: Initialized, Role: FOLLOWER
I20260927 02:42:13.115141 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c2cc88ae3d431aba89b6b35cc2efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } }
I20260927 02:42:13.115214 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:13.115236 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:13.115283 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:13.115841 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c2cc88ae3d431aba89b6b35cc2efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } }
I20260927 02:42:13.115924 998 leader_election.cc:302] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c2cc88ae3d431aba89b6b35cc2efce; no voters:
I20260927 02:42:13.116096 998 leader_election.cc:288] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:13.116137 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:13.116250 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [term 1 LEADER]: Becoming Leader. State: Replica: d6c2cc88ae3d431aba89b6b35cc2efce, State: Running, Role: LEADER
I20260927 02:42:13.116323 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c2cc88ae3d431aba89b6b35cc2efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } }
I20260927 02:42:13.116351 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:13.116665 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c2cc88ae3d431aba89b6b35cc2efce. Latest consensus state: current_term: 1 leader_uuid: "d6c2cc88ae3d431aba89b6b35cc2efce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c2cc88ae3d431aba89b6b35cc2efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } } }
I20260927 02:42:13.116736 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:13.116912 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:13.117780 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:13.116863 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6c2cc88ae3d431aba89b6b35cc2efce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c2cc88ae3d431aba89b6b35cc2efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } } }
I20260927 02:42:13.118556 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:13.119509 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.109289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.0.98.62:41059)
0927 02:42:13.109437 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.109446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.109473 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:42:13.109881 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.109888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.109934 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.110051 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:13.110758 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.111714 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.112721 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.113125 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.113231 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.115818 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:13.115822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:13.115823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:13.115835 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:13.117867 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.118289 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.118292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.118293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.118404 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.118638 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.118639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.118640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.118927 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:13.119075 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.119190 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.119320 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260927 02:42:13.119656 1008 catalog_manager.cc:1347] Generated new cluster ID: b848625bbb0648aba4334736c37b39e1
I20260927 02:42:13.119694 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:13.124442 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:13.125043 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:13.130939 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c2cc88ae3d431aba89b6b35cc2efce: Generated new TSK 0
I20260927 02:42:13.131171 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:13.162958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41059
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:13.243769 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:13.243975 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:13.244006 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:13.245225 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:13.245268 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:13.245303 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:13.246732 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41059
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:13.246964 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:13.247165 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:13.252140 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:13.252140 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:13.252223 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:13.253425 1024 server_base.cc:1034] running on GCE node
I20260927 02:42:13.253640 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:13.253875 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:13.255028 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790476933255018 us; error 29 us; skew 500 ppm
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:13.257771 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:13.258239 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45407/ using document root <none> and password file <none>
I20260927 02:42:13.258481 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:13.258528 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:13.258636 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:13.259693 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "34a34fb2e31f4e35bc906f3473549a80"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "3785bd610fb6aec3b83263171dbb12e7"
server_key_iv: "a67c8ad1a4b054889abdf558f9d7dfe8"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.260015 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "34a34fb2e31f4e35bc906f3473549a80"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "3785bd610fb6aec3b83263171dbb12e7"
server_key_iv: "a67c8ad1a4b054889abdf558f9d7dfe8"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.261423 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:13.262249 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.262421 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:13.262543 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "34a34fb2e31f4e35bc906f3473549a80"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "3785bd610fb6aec3b83263171dbb12e7"
server_key_iv: "a67c8ad1a4b054889abdf558f9d7dfe8"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.262619 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:13.275887 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:13.278095 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:13.278208 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:13.278394 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:13.278678 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:13.278712 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.278733 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:13.278749 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.284934 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42907
I20260927 02:42:13.284970 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42907 every 8 connection(s)
I20260927 02:42:13.285316 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:13.287954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 1024
I20260927 02:42:13.288046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:13.288225 392 external_mini_cluster.cc:1428] Setting key 1daf974b259c84e99218493d379138cd
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:13.296134 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.286176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41049 (local address 127.0.98.62:41059)
0927 02:42:13.286235 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.286238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.286419 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:42:13.286948 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.286950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.286962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.287017 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:13.287980 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.288709 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.290063 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.290420 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.290502 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.293114 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:13.293115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:13.293116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:13.293123 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:13.294485 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.295006 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.295007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.295008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.295069 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.295379 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.295380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.295380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.295501 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:13.295557 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.295923 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.296004 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:42:13.296310 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.286058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41059 (local address 127.0.98.1:41049)
0927 02:42:13.286286 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:13.286336 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:13.286850 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:13.287091 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:13.287095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:13.287678 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:13.287882 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.287888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.288791 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.288793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.289966 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.289969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.290555 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.290556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.290681 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.291295 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:13.291306 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:13.292982 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:13.294573 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:13.294581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:13.294583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:13.294900 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:13.295137 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:13.295138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:13.295138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:13.295286 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:13.295565 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:13.295572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:13.295749 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 02:42:13.295875 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 02:42:13.296057 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:42:13.297138 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41059
I20260927 02:42:13.297245 1153 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:13.297399 1153 heartbeater.cc:511] Master 127.0.98.62:41059 requested a full tablet report, sending...
I20260927 02:42:13.297966 962 ts_manager.cc:194] Registered new tserver with Master: 34a34fb2e31f4e35bc906f3473549a80 (127.0.98.1:42907)
I20260927 02:42:13.298578 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:13.315510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41059
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:13.395803 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:13.396024 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:13.396049 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:13.397265 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:13.397313 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:13.397341 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:13.398846 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41059
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:13.399058 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:13.399237 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:13.404477 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:13.404472 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:13.404477 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:13.404775 1158 server_base.cc:1034] running on GCE node
I20260927 02:42:13.404973 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:13.405190 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:13.406347 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790476933406317 us; error 44 us; skew 500 ppm
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:13.409145 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:13.409690 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35719/ using document root <none> and password file <none>
I20260927 02:42:13.409922 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:13.409970 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:13.410102 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:13.411139 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "465902f74706418fbb6efd0f84604a50"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "3e9dbf6d4c8ddc13d844f0a9843670a3"
server_key_iv: "cf50c510ab68577767b76586353a3f5b"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.411474 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "465902f74706418fbb6efd0f84604a50"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "3e9dbf6d4c8ddc13d844f0a9843670a3"
server_key_iv: "cf50c510ab68577767b76586353a3f5b"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.412846 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:13.413707 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.413923 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:13.414017 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "465902f74706418fbb6efd0f84604a50"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "3e9dbf6d4c8ddc13d844f0a9843670a3"
server_key_iv: "cf50c510ab68577767b76586353a3f5b"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.414084 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:13.439540 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:13.441977 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:13.442097 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:13.442346 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:13.442653 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:13.442683 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.442708 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:13.442728 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.448333 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40917
I20260927 02:42:13.448441 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40917 every 8 connection(s)
I20260927 02:42:13.448745 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:13.450810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 1158
I20260927 02:42:13.450888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:13.451179 392 external_mini_cluster.cc:1428] Setting key 14b7954766a7f639f26eda83ae1c5a89
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:42:13.459410 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.449550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48319 (local address 127.0.98.62:41059)
0927 02:42:13.449680 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.449682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.449780 (+ 98us) server_negotiation.cc:407] Connection header received
0927 02:42:13.450359 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.450361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.450373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.450421 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:13.451561 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.452130 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.453562 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.453769 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.453827 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.456317 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:13.456320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:13.456320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:13.456329 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:13.457944 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.458409 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.458411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.458411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.458472 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.458722 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.458723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.458724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.458850 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:13.458906 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.459224 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.459313 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 02:42:13.459591 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.449463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41059 (local address 127.0.98.2:48319)
0927 02:42:13.449639 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:13.449691 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:13.450258 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:13.450510 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:13.450513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:13.451151 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:13.451397 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.451405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.452233 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.452234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.453475 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.453478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.453853 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.453853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.453987 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.454500 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:13.454509 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:13.456149 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:13.458053 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:13.458063 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:13.458066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:13.458317 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:13.458532 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:13.458532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:13.458533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:13.458653 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:13.458904 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:13.458912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:13.459086 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 02:42:13.459181 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:42:13.459358 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260927 02:42:13.460189 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41059
I20260927 02:42:13.460295 1287 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:13.460491 1287 heartbeater.cc:511] Master 127.0.98.62:41059 requested a full tablet report, sending...
I20260927 02:42:13.460969 962 ts_manager.cc:194] Registered new tserver with Master: 465902f74706418fbb6efd0f84604a50 (127.0.98.2:40917)
I20260927 02:42:13.461407 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:13.480170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41059
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:13.559911 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:13.560130 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:13.560163 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:13.561396 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:13.561446 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:13.561482 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:13.562932 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41059
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:13.563174 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:13.563395 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:13.568262 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:13.568331 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:13.568289 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:13.568988 1292 server_base.cc:1034] running on GCE node
I20260927 02:42:13.569288 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:13.569525 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:13.570686 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790476933570669 us; error 31 us; skew 500 ppm
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:13.573246 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:13.573701 1292 webserver.cc:460] Webserver started at http://127.0.98.3:41383/ using document root <none> and password file <none>
I20260927 02:42:13.573911 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:13.573954 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:13.574040 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:13.575258 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "c3f66f95b576437aa84407dbb9ac576a"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "3962e384c5ad8fe9ef85c11fbbea671f"
server_key_iv: "139ad858ccb545f9bc678e2fcec1cf68"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.575560 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c3f66f95b576437aa84407dbb9ac576a"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "3962e384c5ad8fe9ef85c11fbbea671f"
server_key_iv: "139ad858ccb545f9bc678e2fcec1cf68"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.576957 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:13.577811 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.578053 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:13.578145 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "c3f66f95b576437aa84407dbb9ac576a"
format_stamp: "Formatted at 2026-09-27 02:42:13 on dist-test-slave-44kr"
server_key: "3962e384c5ad8fe9ef85c11fbbea671f"
server_key_iv: "139ad858ccb545f9bc678e2fcec1cf68"
server_key_version: "encryptionkey@0"
I20260927 02:42:13.578224 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:13.593712 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:13.595870 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:13.596011 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:13.596232 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:13.596565 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:13.596607 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.596638 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:13.596661 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:13.601933 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43337
I20260927 02:42:13.602010 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43337 every 8 connection(s)
I20260927 02:42:13.602365 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:13.604997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 1292
I20260927 02:42:13.605104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:13.605257 392 external_mini_cluster.cc:1428] Setting key 1348c9aeef87a5c3c5afeb3591c04d35
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:13.612640 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.603145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34703 (local address 127.0.98.62:41059)
0927 02:42:13.603224 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.603227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.603392 (+ 165us) server_negotiation.cc:407] Connection header received
0927 02:42:13.603944 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.603945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.603957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.604017 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:13.605052 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.605697 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.607077 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.607289 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.607351 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.609665 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:13.609667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:13.609667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:13.609674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:13.610988 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.611447 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.611448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.611449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.611511 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.611799 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.611800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.611800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.611988 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:13.612124 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.612487 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.612565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:42:13.612762 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.603060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41059 (local address 127.0.98.3:34703)
0927 02:42:13.603266 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:13.603309 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:13.603831 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:13.604088 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:13.604092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:13.604700 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:13.604937 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.604944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.605844 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.605845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.606920 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.606924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.607392 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.607393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.607519 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.608014 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:13.608023 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:13.609563 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:13.611073 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:13.611079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:13.611081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:13.611349 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:13.611602 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:13.611603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:13.611603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:13.611718 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:13.612159 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:13.612165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:13.612294 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 02:42:13.612425 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 02:42:13.612573 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":123,"threads_started":1}
I20260927 02:42:13.613483 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41059
I20260927 02:42:13.613577 1421 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:13.613780 1421 heartbeater.cc:511] Master 127.0.98.62:41059 requested a full tablet report, sending...
I20260927 02:42:13.614241 962 ts_manager.cc:194] Registered new tserver with Master: c3f66f95b576437aa84407dbb9ac576a (127.0.98.3:43337)
I20260927 02:42:13.614616 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34703
I20260927 02:42:13.617486 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:13.625438 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.619196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.0.98.62:41059)
0927 02:42:13.619254 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.619256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.619337 (+ 81us) server_negotiation.cc:407] Connection header received
0927 02:42:13.619392 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.619393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.619403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.619469 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:13.620060 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.620631 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.621379 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.621569 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.621623 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.622878 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:13.622879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:13.622880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:13.622887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:13.624274 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.624626 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.624627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.624628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.624685 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.624893 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.624893 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.624893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.625034 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:13.625105 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.625307 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.625371 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:42:13.627808 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:13.629555 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:13.635906 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.631932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43337 (local address 127.0.0.1:44566)
0927 02:42:13.632093 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:13.632101 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:13.632138 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:13.632445 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:13.632448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:13.632453 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:13.632557 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.632559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.633537 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.633538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.634859 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.634861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.635616 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.635617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.635694 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.635707 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:42:13.635772 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:42:13.635829 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":2,"thread_start_us":36,"threads_started":1}
I20260927 02:42:13.635912 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.632101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.0.98.3:43337)
0927 02:42:13.632259 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.632264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.632273 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:13.632312 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.632314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.632330 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.632426 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:13.632672 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.633448 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.634946 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.635533 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.635610 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.635672 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.635791 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.635830 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260927 02:42:13.636089 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.631748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40917 (local address 127.0.0.1:45382)
0927 02:42:13.631909 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:13.631950 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:13.632010 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:13.632445 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:13.632448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:13.632450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:13.632557 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.632560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.633855 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.633856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.634928 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.634929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.635867 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.635868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.635933 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.635937 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:13.635966 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 02:42:13.636005 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":59,"spinlock_wait_cycles":768,"thread_start_us":55,"threads_started":1}
I20260927 02:42:13.636168 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.631883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.0.98.1:42907)
0927 02:42:13.632551 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.632555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.632562 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:42:13.632600 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.632602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.632616 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.632689 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:13.632900 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.633934 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.634893 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.635730 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.635797 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.635853 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.636046 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.636099 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":87,"threads_started":1}
I20260927 02:42:13.636302 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.631749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42907 (local address 127.0.0.1:43962)
0927 02:42:13.632082 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:13.632093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:13.632151 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:13.632765 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:13.632766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:13.632768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:13.632827 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.632828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.634018 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.634019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.634812 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.634814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.635879 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.635880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.635939 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.635942 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:13.635981 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 02:42:13.636011 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260927 02:42:13.636718 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.631929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.0.98.2:40917)
0927 02:42:13.632260 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.632264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.632273 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:13.632312 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.632314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.632330 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.632426 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:13.632635 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.633745 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.634986 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.635783 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.636497 (+ 714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.636563 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.636600 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.636642 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":76,"threads_started":1}
I20260927 02:42:13.636669 1355 tablet_service.cc:1467] Processing CreateTablet for tablet c6c8c6e03d5c422aae611ebf8c7dd880 (DEFAULT_TABLE table=test-table [id=f2034cbaa36442b29527ea478f89c908]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:13.636997 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6c8c6e03d5c422aae611ebf8c7dd880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:13.636969 1087 tablet_service.cc:1467] Processing CreateTablet for tablet c6c8c6e03d5c422aae611ebf8c7dd880 (DEFAULT_TABLE table=test-table [id=f2034cbaa36442b29527ea478f89c908]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:13.637305 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6c8c6e03d5c422aae611ebf8c7dd880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:13.637264 1221 tablet_service.cc:1467] Processing CreateTablet for tablet c6c8c6e03d5c422aae611ebf8c7dd880 (DEFAULT_TABLE table=test-table [id=f2034cbaa36442b29527ea478f89c908]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:13.637542 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6c8c6e03d5c422aae611ebf8c7dd880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:13.639963 1435 tablet_bootstrap.cc:492] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a: Bootstrap starting.
I20260927 02:42:13.640554 1436 tablet_bootstrap.cc:492] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50: Bootstrap starting.
I20260927 02:42:13.640735 1435 tablet_bootstrap.cc:654] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:13.641168 1436 tablet_bootstrap.cc:654] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:13.641351 1435 log.cc:824] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:13.641680 1436 log.cc:824] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:13.641960 1437 tablet_bootstrap.cc:492] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80: Bootstrap starting.
I20260927 02:42:13.642153 1435 tablet_bootstrap.cc:492] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a: No bootstrap required, opened a new log
I20260927 02:42:13.642223 1435 ts_tablet_manager.cc:1397] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:13.642334 1436 tablet_bootstrap.cc:492] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50: No bootstrap required, opened a new log
I20260927 02:42:13.642397 1436 ts_tablet_manager.cc:1397] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:13.643043 1437 tablet_bootstrap.cc:654] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:13.643658 1437 log.cc:824] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:13.644346 1436 raft_consensus.cc:348] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } }
I20260927 02:42:13.644454 1435 raft_consensus.cc:348] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } }
I20260927 02:42:13.644570 1436 raft_consensus.cc:374] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:13.644613 1436 raft_consensus.cc:729] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465902f74706418fbb6efd0f84604a50, State: Initialized, Role: FOLLOWER
I20260927 02:42:13.644610 1435 raft_consensus.cc:374] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:13.644649 1435 raft_consensus.cc:729] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f66f95b576437aa84407dbb9ac576a, State: Initialized, Role: FOLLOWER
I20260927 02:42:13.644748 1436 consensus_queue.cc:260] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } }
I20260927 02:42:13.644775 1435 consensus_queue.cc:260] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } }
I20260927 02:42:13.644827 1437 tablet_bootstrap.cc:492] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80: No bootstrap required, opened a new log
I20260927 02:42:13.644881 1437 ts_tablet_manager.cc:1397] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:42:13.645007 1436 ts_tablet_manager.cc:1428] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:13.645020 1435 ts_tablet_manager.cc:1428] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:13.645128 1421 heartbeater.cc:503] Master 127.0.98.62:41059 was elected leader, sending a full tablet report...
I20260927 02:42:13.645126 1287 heartbeater.cc:503] Master 127.0.98.62:41059 was elected leader, sending a full tablet report...
I20260927 02:42:13.646557 1437 raft_consensus.cc:348] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } }
I20260927 02:42:13.646698 1437 raft_consensus.cc:374] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:13.646718 1437 raft_consensus.cc:729] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a34fb2e31f4e35bc906f3473549a80, State: Initialized, Role: FOLLOWER
I20260927 02:42:13.646808 1437 consensus_queue.cc:260] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } }
I20260927 02:42:13.647042 1437 ts_tablet_manager.cc:1428] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:13.647145 1153 heartbeater.cc:503] Master 127.0.98.62:41059 was elected leader, sending a full tablet report...
I20260927 02:42:13.671716 1442 raft_consensus.cc:482] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:13.671873 1442 raft_consensus.cc:504] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } }
I20260927 02:42:13.672284 1442 leader_election.cc:288] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 465902f74706418fbb6efd0f84604a50 (127.0.98.2:40917), 34a34fb2e31f4e35bc906f3473549a80 (127.0.98.1:42907)
I20260927 02:42:13.676198 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.672621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43983 (local address 127.0.98.2:40917)
0927 02:42:13.672663 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.672665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.672737 (+ 72us) server_negotiation.cc:407] Connection header received
0927 02:42:13.672765 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.672766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.672777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.672841 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:13.672986 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.673667 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.675157 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.675836 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.675904 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.675913 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.676070 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.676121 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:13.676198 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.672548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41761 (local address 127.0.98.1:42907)
0927 02:42:13.672596 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.672598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.672610 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:42:13.672645 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.672646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.672660 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.672743 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:13.672885 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.673631 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.675158 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.675758 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.675843 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.675857 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.676069 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.676132 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:42:13.676247 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.672484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42907 (local address 127.0.98.3:41761)
0927 02:42:13.672545 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:13.672558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:13.672597 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:13.672748 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:13.672749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:13.672751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:13.672819 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.672820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.673739 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.673741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.675042 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.675045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.675858 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.675859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.675952 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.675957 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:13.676029 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:42:13.676151 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":148}
I20260927 02:42:13.676257 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.672498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40917 (local address 127.0.98.3:43983)
0927 02:42:13.672661 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:13.672669 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:13.672718 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:13.672859 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:13.672859 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:13.672870 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:13.672921 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.672922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.673750 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.673751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.675042 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:13.675045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.675920 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.675921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:13.675975 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.675978 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:13.676029 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:42:13.676143 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":22,"thread_start_us":106,"threads_started":1}
I20260927 02:42:13.676676 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c8c6e03d5c422aae611ebf8c7dd880" candidate_uuid: "c3f66f95b576437aa84407dbb9ac576a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465902f74706418fbb6efd0f84604a50" is_pre_election: true
I20260927 02:42:13.676745 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c8c6e03d5c422aae611ebf8c7dd880" candidate_uuid: "c3f66f95b576437aa84407dbb9ac576a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a34fb2e31f4e35bc906f3473549a80" is_pre_election: true
I20260927 02:42:13.676829 1241 raft_consensus.cc:2454] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f66f95b576437aa84407dbb9ac576a in term 0.
I20260927 02:42:13.676884 1107 raft_consensus.cc:2454] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f66f95b576437aa84407dbb9ac576a in term 0.
I20260927 02:42:13.677060 1311 leader_election.cc:302] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a34fb2e31f4e35bc906f3473549a80, c3f66f95b576437aa84407dbb9ac576a; no voters:
I20260927 02:42:13.677199 1442 raft_consensus.cc:2781] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:13.677279 1442 raft_consensus.cc:482] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:13.677296 1442 raft_consensus.cc:3037] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:13.678032 1442 raft_consensus.cc:504] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } }
I20260927 02:42:13.678161 1442 leader_election.cc:288] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [CANDIDATE]: Term 1 election: Requested vote from peers 465902f74706418fbb6efd0f84604a50 (127.0.98.2:40917), 34a34fb2e31f4e35bc906f3473549a80 (127.0.98.1:42907)
I20260927 02:42:13.678350 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c8c6e03d5c422aae611ebf8c7dd880" candidate_uuid: "c3f66f95b576437aa84407dbb9ac576a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465902f74706418fbb6efd0f84604a50"
I20260927 02:42:13.678407 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c8c6e03d5c422aae611ebf8c7dd880" candidate_uuid: "c3f66f95b576437aa84407dbb9ac576a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a34fb2e31f4e35bc906f3473549a80"
I20260927 02:42:13.678434 1241 raft_consensus.cc:3037] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:13.678478 1107 raft_consensus.cc:3037] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:13.679350 1107 raft_consensus.cc:2454] T c6c8c6e03d5c422aae611ebf8c7dd880 P 34a34fb2e31f4e35bc906f3473549a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f66f95b576437aa84407dbb9ac576a in term 1.
I20260927 02:42:13.679364 1241 raft_consensus.cc:2454] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f66f95b576437aa84407dbb9ac576a in term 1.
I20260927 02:42:13.679579 1311 leader_election.cc:302] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a34fb2e31f4e35bc906f3473549a80, c3f66f95b576437aa84407dbb9ac576a; no voters:
I20260927 02:42:13.679687 1442 raft_consensus.cc:2781] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:13.679857 1442 raft_consensus.cc:686] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [term 1 LEADER]: Becoming Leader. State: Replica: c3f66f95b576437aa84407dbb9ac576a, State: Running, Role: LEADER
I20260927 02:42:13.679946 1442 consensus_queue.cc:237] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } }
I20260927 02:42:13.680805 961 catalog_manager.cc:5495] T c6c8c6e03d5c422aae611ebf8c7dd880 P c3f66f95b576437aa84407dbb9ac576a reported cstate change: term changed from 0 to 1, leader changed from <none> to c3f66f95b576437aa84407dbb9ac576a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c3f66f95b576437aa84407dbb9ac576a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465902f74706418fbb6efd0f84604a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a34fb2e31f4e35bc906f3473549a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3f66f95b576437aa84407dbb9ac576a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43337 } health_report { overall_health: HEALTHY } } }
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:13.689779 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.682571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.98.1:42907)
0927 02:42:13.682639 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.682641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.682725 (+ 84us) server_negotiation.cc:407] Connection header received
0927 02:42:13.682787 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.682788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.682799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.682836 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:13.683350 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.683825 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.684572 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.684798 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.684850 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.687222 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:13.687234 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:13.687234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:13.687241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:13.688722 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.689028 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.689032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.689032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.689086 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.689250 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.689251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.689251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.689398 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:13.689533 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.689612 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.689686 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 02:42:13.699308 1288 tablet.cc:2369] T c6c8c6e03d5c422aae611ebf8c7dd880 P 465902f74706418fbb6efd0f84604a50: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:13.712242 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.705109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.0.98.1:42907)
0927 02:42:13.705186 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.705187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.705256 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:42:13.705282 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.705283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.705292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.705337 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:13.705951 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.706484 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.707284 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.707478 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.707540 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.709936 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:13.709938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:13.709938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:13.709945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:13.711214 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.711516 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.711518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.711518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.711567 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.711723 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.711724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.711724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.711819 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:13.711903 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.712079 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.712153 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:13.733824 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:13.726899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.0.98.1:42907)
0927 02:42:13.726946 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:13.726949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:13.727025 (+ 76us) server_negotiation.cc:407] Connection header received
0927 02:42:13.727078 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:13.727079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:13.727088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:13.727135 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:13.727626 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.728106 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.728934 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:13.729144 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:13.729206 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:13.731424 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:13.731425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:13.731426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:13.731432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:13.732879 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.733182 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.733183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.733183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.733268 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:13.733409 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:13.733410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:13.733410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:13.733494 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:13.733557 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:13.733705 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 02:42:13.733757 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 02:42:13.734254 1067 tablet_service.cc:2908] Rejecting scan request for tablet c6c8c6e03d5c422aae611ebf8c7dd880: Uninitialized: clean time has not yet been initialized
I20260927 02:42:13.735116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 1024
I20260927 02:42:13.739610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 1158
I20260927 02:42:13.743530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 1292
I20260927 02:42:13.747392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 930
2026-09-27T02:42:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1840 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[1459](info): set up 2 sockets
Sep 27 02:42:13 dist-test-slave-44kr krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260927 02:42:14.715816 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:14 dist-test-slave-44kr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:14.835506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:14.915725 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:14.915934 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:14.915956 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:14.917160 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:14.917201 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:42:14.917213 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:14.917225 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:14.917236 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:14.917248 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:14.918735 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35795
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:14.918959 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:14.919171 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:14.923960 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:14.923969 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:14.924006 1472 server_base.cc:1034] running on GCE node
W20260927 02:42:14.923976 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:14.924515 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:14.924806 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:14.925968 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790476934925949 us; error 35 us; skew 500 ppm
Sep 27 02:42:14 dist-test-slave-44kr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:14.928603 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:14.929061 1472 webserver.cc:460] Webserver started at http://127.0.98.62:35835/ using document root <none> and password file <none>
I20260927 02:42:14.929288 1472 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:14.929333 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:14.929451 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:14.930517 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "1648330d718646c09c1df5a2f718191b"
format_stamp: "Formatted at 2026-09-27 02:42:14 on dist-test-slave-44kr"
server_key: "01768add80ef7178c1f9b618377ea1b2"
server_key_iv: "cf3aab5156c88e61827f95711c0bb192"
server_key_version: "encryptionkey@0"
I20260927 02:42:14.930836 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "1648330d718646c09c1df5a2f718191b"
format_stamp: "Formatted at 2026-09-27 02:42:14 on dist-test-slave-44kr"
server_key: "01768add80ef7178c1f9b618377ea1b2"
server_key_iv: "cf3aab5156c88e61827f95711c0bb192"
server_key_version: "encryptionkey@0"
I20260927 02:42:14.932149 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:14.933075 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:14.933257 1472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:14.933346 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "1648330d718646c09c1df5a2f718191b"
format_stamp: "Formatted at 2026-09-27 02:42:14 on dist-test-slave-44kr"
server_key: "01768add80ef7178c1f9b618377ea1b2"
server_key_iv: "cf3aab5156c88e61827f95711c0bb192"
server_key_version: "encryptionkey@0"
I20260927 02:42:14.933408 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:14.959933 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:14.962318 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:14.962460 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:14.966928 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35795
I20260927 02:42:14.966987 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35795 every 8 connection(s)
I20260927 02:42:14.967306 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:14.967942 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:14.970168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 1472
I20260927 02:42:14.970206 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b: Bootstrap starting.
I20260927 02:42:14.970252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:14.970374 392 external_mini_cluster.cc:1428] Setting key 2b5ca0f7aac55b52ebd39c321d548b98
I20260927 02:42:14.970870 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:14.971185 1550 log.cc:824] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:14.971915 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b: No bootstrap required, opened a new log
I20260927 02:42:14.974133 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1648330d718646c09c1df5a2f718191b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } }
I20260927 02:42:14.974277 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:14.974298 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1648330d718646c09c1df5a2f718191b, State: Initialized, Role: FOLLOWER
I20260927 02:42:14.974408 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1648330d718646c09c1df5a2f718191b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } }
I20260927 02:42:14.974478 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:14.974499 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:14.974543 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:14.975184 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1648330d718646c09c1df5a2f718191b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } }
I20260927 02:42:14.975291 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1648330d718646c09c1df5a2f718191b; no voters:
I20260927 02:42:14.975442 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:14.975481 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:14.975630 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [term 1 LEADER]: Becoming Leader. State: Replica: 1648330d718646c09c1df5a2f718191b, State: Running, Role: LEADER
I20260927 02:42:14.975694 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:14.975723 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1648330d718646c09c1df5a2f718191b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } }
I20260927 02:42:14.976042 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1648330d718646c09c1df5a2f718191b. Latest consensus state: current_term: 1 leader_uuid: "1648330d718646c09c1df5a2f718191b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1648330d718646c09c1df5a2f718191b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } } }
I20260927 02:42:14.976125 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:14.976507 1561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:14.976994 1561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:14.977329 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1648330d718646c09c1df5a2f718191b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1648330d718646c09c1df5a2f718191b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } } }
I20260927 02:42:14.977399 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b [sys.catalog]: This master's current role is: LEADER
Sep 27 02:42:14 dist-test-slave-44kr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:14.979292 1561 catalog_manager.cc:1347] Generated new cluster ID: 3cfb90e933134a7fbb7298867d2fef45
I20260927 02:42:14.979343 1561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:14.982369 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:14.970897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.0.98.62:35795)
0927 02:42:14.971065 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:14.971074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:14.971108 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:42:14.971582 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:14.971592 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:14.971656 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:14.971804 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:14.972426 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:14.973811 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:14.974638 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:14.975258 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:14.975360 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:14.978251 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:14.978255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:14.978256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:14.978268 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:14.980317 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:14.980784 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:14.980788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:14.980788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:14.980892 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:14.981088 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:14.981089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:14.981090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:14.981338 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:14.981488 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:14.981824 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 02:42:14.981971 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260927 02:42:14.997484 1561 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:14.998163 1561 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:15.002432 1561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b: Generated new TSK 0
I20260927 02:42:15.002646 1561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:15.035753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35795
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:15.134903 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:15.135187 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:15.135217 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:15.136837 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:15.136902 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:15.136924 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:42:15.136942 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:15.138836 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:15.139104 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:15.139351 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:42:15.144758 1576 server_base.cc:1034] running on GCE node
W20260927 02:42:15.144769 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:15.144963 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:15.144790 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:15.145215 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:15.145469 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:15.146641 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790476935146621 us; error 37 us; skew 500 ppm
Sep 27 02:42:15 dist-test-slave-44kr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:15.149294 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:15.149735 1576 webserver.cc:460] Webserver started at http://127.0.98.1:37035/ using document root <none> and password file <none>
I20260927 02:42:15.149955 1576 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:15.150003 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:15.150137 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:15.151208 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "3273e35dae6245c68789c144902a594d"
format_stamp: "Formatted at 2026-09-27 02:42:15 on dist-test-slave-44kr"
server_key: "030f24e77c2d2e8017ef66dda09cfd53"
server_key_iv: "01a58d71f52da05a69b4b34051633d2a"
server_key_version: "encryptionkey@0"
I20260927 02:42:15.151546 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3273e35dae6245c68789c144902a594d"
format_stamp: "Formatted at 2026-09-27 02:42:15 on dist-test-slave-44kr"
server_key: "030f24e77c2d2e8017ef66dda09cfd53"
server_key_iv: "01a58d71f52da05a69b4b34051633d2a"
server_key_version: "encryptionkey@0"
I20260927 02:42:15.153738 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 02:42:15.154543 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:15.154736 1576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:15.154836 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "3273e35dae6245c68789c144902a594d"
format_stamp: "Formatted at 2026-09-27 02:42:15 on dist-test-slave-44kr"
server_key: "030f24e77c2d2e8017ef66dda09cfd53"
server_key_iv: "01a58d71f52da05a69b4b34051633d2a"
server_key_version: "encryptionkey@0"
I20260927 02:42:15.154908 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:15.164337 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:15.166442 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:15.166576 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:15.167052 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:15.167102 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:15.167136 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:15.167161 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:15.173753 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45897
I20260927 02:42:15.173868 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45897 every 8 connection(s)
I20260927 02:42:15.174175 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:42:15 dist-test-slave-44kr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:15.177819 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.167601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46829 (local address 127.0.98.62:35795)
0927 02:42:15.167647 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.167649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.167917 (+ 268us) server_negotiation.cc:407] Connection header received
0927 02:42:15.168538 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.168539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.168552 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.168605 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:15.169474 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.170261 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.171841 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.172110 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.172168 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.174956 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:15.174958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:15.174958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:15.174968 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:15.176454 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.176838 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.176839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.176839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.176896 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.177140 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.177141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.177141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.177310 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:15.177379 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.177658 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.177748 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:42:15.177963 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.167623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35795 (local address 127.0.98.1:46829)
0927 02:42:15.167789 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:15.167825 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:15.168424 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:15.168733 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:15.168738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:15.169090 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:15.169388 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.169395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.170368 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.170369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.171744 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.171748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.172203 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.172204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.172353 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.173030 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:15.173042 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:15.174820 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:15.176515 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.176521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.176523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.176783 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.176939 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.176940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.176940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.177037 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.177378 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:15.177383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:15.177516 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 02:42:15.177638 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 02:42:15.177773 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260927 02:42:15.178503 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35795
I20260927 02:42:15.178597 1707 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:15.178824 1707 heartbeater.cc:511] Master 127.0.98.62:35795 requested a full tablet report, sending...
I20260927 02:42:15.179481 1503 ts_manager.cc:194] Registered new tserver with Master: 3273e35dae6245c68789c144902a594d (127.0.98.1:45897)
I20260927 02:42:15.180371 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46829
I20260927 02:42:15.181447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 1576
I20260927 02:42:15.181535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:15.181720 392 external_mini_cluster.cc:1428] Setting key 29250ecd560704aa3dc54cf78ab6d779
I20260927 02:42:15.185427 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.179560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35795 (local address 127.0.98.1:57051)
0927 02:42:15.179738 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:15.179750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:15.179816 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:15.179928 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:15.179929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:15.180124 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:15.180193 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.180194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.181005 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.181007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.181680 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.181683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.182064 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.182065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.182129 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.182567 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:15.182572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:15.182880 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:15.184487 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.184488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.184488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.184656 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.184851 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.184852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.184852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.184907 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.185164 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:15.185165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:15.185206 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:42:15.185269 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:42:15.185355 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260927 02:42:15.185459 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.179632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57051 (local address 127.0.98.62:35795)
0927 02:42:15.179678 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.179680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.179833 (+ 153us) server_negotiation.cc:407] Connection header received
0927 02:42:15.179858 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.179859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.179868 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.179916 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:15.180285 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.180914 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.181764 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.182007 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.182065 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.182962 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:15.182963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:15.182963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:15.182970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:15.184345 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.184726 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.184727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.184728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.184789 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.184960 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.184961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.184961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.185079 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:15.185142 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.185299 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.185372 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:15.208160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35795
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:15.288794 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:15.289000 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:15.289021 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:15.290120 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:15.290156 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:15.290174 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:42:15.290190 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:15.291658 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:15.291891 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:15.292080 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:15.297043 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:15.297165 1716 server_base.cc:1034] running on GCE node
W20260927 02:42:15.297184 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:15.297372 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:15.297566 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:15.297811 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:15.298976 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790476935298960 us; error 29 us; skew 500 ppm
Sep 27 02:42:15 dist-test-slave-44kr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:15.301775 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:15.302309 1716 webserver.cc:460] Webserver started at http://127.0.98.2:43283/ using document root <none> and password file <none>
I20260927 02:42:15.302531 1716 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:15.302582 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:15.302717 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:15.303776 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c6150b5383c4a51b88482c579c118a3"
format_stamp: "Formatted at 2026-09-27 02:42:15 on dist-test-slave-44kr"
server_key: "452ee4a5892a446c648746ee4e6f3fa7"
server_key_iv: "2061bd00a98cc0f5f9fdff8f0065cad5"
server_key_version: "encryptionkey@0"
I20260927 02:42:15.304122 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c6150b5383c4a51b88482c579c118a3"
format_stamp: "Formatted at 2026-09-27 02:42:15 on dist-test-slave-44kr"
server_key: "452ee4a5892a446c648746ee4e6f3fa7"
server_key_iv: "2061bd00a98cc0f5f9fdff8f0065cad5"
server_key_version: "encryptionkey@0"
I20260927 02:42:15.305539 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:42:15.306403 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:15.306612 1716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:15.306702 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "2c6150b5383c4a51b88482c579c118a3"
format_stamp: "Formatted at 2026-09-27 02:42:15 on dist-test-slave-44kr"
server_key: "452ee4a5892a446c648746ee4e6f3fa7"
server_key_iv: "2061bd00a98cc0f5f9fdff8f0065cad5"
server_key_version: "encryptionkey@0"
I20260927 02:42:15.306770 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:15.315492 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:15.317643 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:15.317777 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:15.318249 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:15.318285 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:15.318321 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:15.318346 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:15.324988 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42985
Sep 27 02:42:15 dist-test-slave-44kr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:15.325361 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:15.325943 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42985 every 8 connection(s)
I20260927 02:42:15.329039 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.318742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53777 (local address 127.0.98.62:35795)
0927 02:42:15.318791 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.318793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.319007 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:42:15.319633 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.319634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.319648 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.319709 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:15.320610 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.321349 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.322851 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.323145 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.323197 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.326084 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:15.326085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:15.326086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:15.326094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:15.327615 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.328068 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.328069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.328070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.328133 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.328355 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.328355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.328356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.328515 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:15.328582 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.328860 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.328941 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:42:15.329130 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.318750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35795 (local address 127.0.98.2:53777)
0927 02:42:15.318899 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:15.318933 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:15.319503 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:15.319825 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:15.319830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:15.320169 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:15.320507 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.320514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.321438 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.321439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.322757 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.322760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.323215 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.323215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.323363 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.323969 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:15.323986 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:15.325764 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:15.327744 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.327752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.327754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.327994 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.328196 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.328197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.328197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.328300 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.328591 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:15.328598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:15.328736 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 02:42:15.328828 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:42:15.328954 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260927 02:42:15.329891 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35795
I20260927 02:42:15.330006 1847 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:15.330204 1847 heartbeater.cc:511] Master 127.0.98.62:35795 requested a full tablet report, sending...
I20260927 02:42:15.330586 1504 ts_manager.cc:194] Registered new tserver with Master: 2c6150b5383c4a51b88482c579c118a3 (127.0.98.2:42985)
I20260927 02:42:15.331027 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53777
I20260927 02:42:15.332918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 1716
I20260927 02:42:15.333012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:15.333200 392 external_mini_cluster.cc:1428] Setting key 6f04ce8fa3006e464ead6cc46445158d
I20260927 02:42:15.336767 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.330859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60545 (local address 127.0.98.62:35795)
0927 02:42:15.330897 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.330898 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.330903 (+ 5us) server_negotiation.cc:407] Connection header received
0927 02:42:15.330919 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.330920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.330928 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.330968 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:15.331306 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.331778 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.332462 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.332642 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.332702 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.333873 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:15.333874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:15.333875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:15.333881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:15.335458 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.335793 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.335794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.335795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.335855 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.336114 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.336115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.336115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.336290 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:15.336347 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.336634 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.336704 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:42:15.336830 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.330668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35795 (local address 127.0.98.2:60545)
0927 02:42:15.330779 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:15.330788 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:15.330830 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:15.330986 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:15.330987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:15.331182 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:15.331243 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.331244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.331865 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.331866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.332379 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.332380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.332702 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.332702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.332786 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.333319 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:15.333324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:15.333768 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:15.335556 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.335557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.335558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.335728 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.335959 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.335960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.335961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.336035 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.336364 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:15.336365 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:15.336474 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 02:42:15.336608 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 02:42:15.336721 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260927 02:42:15.685191 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.351s user 0.000s sys 0.003s
I20260927 02:42:15.685503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35795
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:15.765766 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:15.765976 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:15.766000 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:15.767174 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:15.767216 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:15.767236 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:42:15.767252 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:15.768738 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:15.768952 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:15.769241 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:15.774125 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:15.774075 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:15.774076 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:15.774241 1856 server_base.cc:1034] running on GCE node
I20260927 02:42:15.774621 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:15.774878 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:15.776091 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790476935776015 us; error 27 us; skew 500 ppm
Sep 27 02:42:15 dist-test-slave-44kr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:15.778924 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:15.779428 1856 webserver.cc:460] Webserver started at http://127.0.98.3:41613/ using document root <none> and password file <none>
I20260927 02:42:15.779649 1856 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:15.779701 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:15.779834 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:15.781014 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "abaf18ee0752429da7894a08cec63fe0"
format_stamp: "Formatted at 2026-09-27 02:42:15 on dist-test-slave-44kr"
server_key: "b8da46623823a91c9c969a2bbe861681"
server_key_iv: "d9f4ab1cd31884c2a7221c64fea02ace"
server_key_version: "encryptionkey@0"
I20260927 02:42:15.781368 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "abaf18ee0752429da7894a08cec63fe0"
format_stamp: "Formatted at 2026-09-27 02:42:15 on dist-test-slave-44kr"
server_key: "b8da46623823a91c9c969a2bbe861681"
server_key_iv: "d9f4ab1cd31884c2a7221c64fea02ace"
server_key_version: "encryptionkey@0"
I20260927 02:42:15.782727 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:42:15.783557 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:15.783859 1856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:15.783954 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "abaf18ee0752429da7894a08cec63fe0"
format_stamp: "Formatted at 2026-09-27 02:42:15 on dist-test-slave-44kr"
server_key: "b8da46623823a91c9c969a2bbe861681"
server_key_iv: "d9f4ab1cd31884c2a7221c64fea02ace"
server_key_version: "encryptionkey@0"
I20260927 02:42:15.784032 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:15.805346 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:15.807451 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:15.807601 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:15.808133 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:15.808171 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:15.808202 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:15.808218 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:15.815040 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40103
Sep 27 02:42:15 dist-test-slave-44kr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:15.815244 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40103 every 8 connection(s)
I20260927 02:42:15.815470 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:15.818291 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.808717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42545 (local address 127.0.98.62:35795)
0927 02:42:15.808780 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.808782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.808976 (+ 194us) server_negotiation.cc:407] Connection header received
0927 02:42:15.809524 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.809525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.809538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.809588 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:15.810401 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.811163 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.812668 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.812972 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.813067 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.815688 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:15.815689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:15.815690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:15.815698 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:15.817005 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.817389 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.817390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.817390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.817446 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.817668 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.817669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.817670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.817790 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:15.817844 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.818113 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.818199 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:42:15.818481 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.808696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35795 (local address 127.0.98.3:42545)
0927 02:42:15.808850 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:15.808892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:15.809384 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:15.809670 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:15.809675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:15.810021 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:15.810306 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.810312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.811243 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.811244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.812553 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.812556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.813086 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.813087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.813241 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.813812 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:15.813823 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:15.815585 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:15.817083 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.817090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.817093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.817321 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.817498 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.817499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.817499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.817610 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.817843 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:15.817849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:15.817995 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 02:42:15.818087 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:42:15.818260 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260927 02:42:15.818907 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35795
I20260927 02:42:15.818992 1987 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:15.819175 1987 heartbeater.cc:511] Master 127.0.98.62:35795 requested a full tablet report, sending...
I20260927 02:42:15.819530 1503 ts_manager.cc:194] Registered new tserver with Master: abaf18ee0752429da7894a08cec63fe0 (127.0.98.3:40103)
I20260927 02:42:15.820031 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42545
I20260927 02:42:15.820561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 1856
I20260927 02:42:15.820654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:15.820807 392 external_mini_cluster.cc:1428] Setting key 92f06c4812098336b6bcb00194ac3cab
I20260927 02:42:15.821535 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:15.825719 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.820080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51337 (local address 127.0.98.62:35795)
0927 02:42:15.820119 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.820121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.820188 (+ 67us) server_negotiation.cc:407] Connection header received
0927 02:42:15.820240 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.820241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.820252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.820325 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:15.820740 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.821285 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.821979 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.822243 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.822310 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.823316 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:15.823316 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:15.823317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:15.823322 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:15.824744 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.825044 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.825045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.825045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.825102 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.825261 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.825262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.825262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.825370 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:15.825423 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.825560 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.825625 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:42:15.825736 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.820007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35795 (local address 127.0.98.3:51337)
0927 02:42:15.820126 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:15.820144 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:15.820195 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:15.820332 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:15.820332 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:15.820566 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:15.820628 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.820630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.821373 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.821374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.821909 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.821911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.822340 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.822341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.822399 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.822851 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:15.822856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:15.823234 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:15.824820 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.824822 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.824822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.824984 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.825154 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:15.825155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:15.825155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:15.825213 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:15.825420 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:15.825421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:15.825464 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:42:15.825523 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:42:15.825627 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
Sep 27 02:42:15 dist-test-slave-44kr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:42:15 dist-test-slave-44kr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:15.851879 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.844756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.0.98.62:35795)
0927 02:42:15.844826 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.844828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.844837 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:15.844868 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.844869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.844880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.844956 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:15.845718 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.846252 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.847273 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.847484 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.847536 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.849663 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:15.849664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:15.849665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:15.849673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:15.850906 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.851212 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.851214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.851214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.851266 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:15.851424 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:15.851425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:15.851425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:15.851531 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:15.851622 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.851716 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.851788 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:42:15.853534 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:15.855075 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:15.861495 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.857372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42985 (local address 127.0.0.1:35700)
0927 02:42:15.857709 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:15.857753 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:15.857820 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:15.858152 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:15.858153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:15.858154 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:15.858219 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.858221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.859478 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.859479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.860425 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.860427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.861191 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.861192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.861269 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.861285 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:42:15.861353 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:42:15.861405 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":30,"thread_start_us":49,"threads_started":1}
I20260927 02:42:15.861502 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.857473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.0.98.2:42985)
0927 02:42:15.857683 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.857688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.857899 (+ 211us) server_negotiation.cc:407] Connection header received
0927 02:42:15.857931 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.857933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.857947 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.858101 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:15.858327 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.859390 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.860504 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.861106 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.861179 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.861248 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.861373 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.861415 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":56,"threads_started":1}
I20260927 02:42:15.862025 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.857561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40103 (local address 127.0.0.1:53042)
0927 02:42:15.858130 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:15.858139 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:15.858187 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:15.858403 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:15.858404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:15.858405 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:15.858441 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.858442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.859371 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.859372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.860733 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.860735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.861801 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.861802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.861864 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.861868 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:15.861914 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:42:15.861955 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":64,"thread_start_us":90,"threads_started":1}
I20260927 02:42:15.862170 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.857547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.0.98.1:45897)
0927 02:42:15.857865 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.857869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.857880 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:15.857915 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.857917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.857933 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.858015 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:15.858283 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.859380 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.861145 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.861862 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.861927 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.861980 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.862063 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.862102 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":61,"threads_started":1}
I20260927 02:42:15.862344 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.857384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45897 (local address 127.0.0.1:43036)
0927 02:42:15.857728 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:15.857753 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:15.857818 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:15.858033 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:15.858036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:15.858038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:15.858105 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.858107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.859495 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.859496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.861012 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:15.861014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.861943 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.861943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:15.862005 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.862008 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:15.862051 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:42:15.862174 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":41,"thread_start_us":74,"threads_started":1}
I20260927 02:42:15.862414 1781 tablet_service.cc:1467] Processing CreateTablet for tablet b254ac1b78ef4048bc61c910cab76322 (DEFAULT_TABLE table=test-table [id=ac616b69df9347a89d7161f9be910e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:15.862581 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:15.857661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.0.98.3:40103)
0927 02:42:15.857827 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:15.857831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:15.858212 (+ 381us) server_negotiation.cc:407] Connection header received
0927 02:42:15.858255 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:15.858257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:15.858273 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:15.858340 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:15.858504 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.859286 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.860827 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:15.861604 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:15.861681 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:15.861754 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:15.862441 (+ 687us) server_negotiation.cc:299] Negotiation successful
0927 02:42:15.862485 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260927 02:42:15.862805 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b254ac1b78ef4048bc61c910cab76322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:15.863139 1921 tablet_service.cc:1467] Processing CreateTablet for tablet b254ac1b78ef4048bc61c910cab76322 (DEFAULT_TABLE table=test-table [id=ac616b69df9347a89d7161f9be910e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:15.863454 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b254ac1b78ef4048bc61c910cab76322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:15.864848 1641 tablet_service.cc:1467] Processing CreateTablet for tablet b254ac1b78ef4048bc61c910cab76322 (DEFAULT_TABLE table=test-table [id=ac616b69df9347a89d7161f9be910e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:15.865196 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b254ac1b78ef4048bc61c910cab76322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:15.865294 2010 tablet_bootstrap.cc:492] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: Bootstrap starting.
I20260927 02:42:15.865866 2010 tablet_bootstrap.cc:654] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:15.866122 2010 log.cc:824] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:15.866725 2012 tablet_bootstrap.cc:492] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: Bootstrap starting.
I20260927 02:42:15.866770 2010 tablet_bootstrap.cc:492] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: No bootstrap required, opened a new log
I20260927 02:42:15.866823 2010 ts_tablet_manager.cc:1397] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:15.867377 2012 tablet_bootstrap.cc:654] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:15.867717 2012 log.cc:824] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:15.867971 2013 tablet_bootstrap.cc:492] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: Bootstrap starting.
I20260927 02:42:15.868592 2013 tablet_bootstrap.cc:654] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:15.868849 2013 log.cc:824] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:15.868979 2012 tablet_bootstrap.cc:492] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: No bootstrap required, opened a new log
I20260927 02:42:15.869061 2012 ts_tablet_manager.cc:1397] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:15.869463 2013 tablet_bootstrap.cc:492] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: No bootstrap required, opened a new log
I20260927 02:42:15.869520 2013 ts_tablet_manager.cc:1397] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:15.869509 2010 raft_consensus.cc:348] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } }
I20260927 02:42:15.869688 2010 raft_consensus.cc:374] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:15.869729 2010 raft_consensus.cc:729] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6150b5383c4a51b88482c579c118a3, State: Initialized, Role: FOLLOWER
I20260927 02:42:15.869889 2010 consensus_queue.cc:260] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } }
I20260927 02:42:15.870141 2010 ts_tablet_manager.cc:1428] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:15.870265 1847 heartbeater.cc:503] Master 127.0.98.62:35795 was elected leader, sending a full tablet report...
I20260927 02:42:15.870869 2012 raft_consensus.cc:348] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } }
I20260927 02:42:15.871013 2012 raft_consensus.cc:374] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:15.871054 2012 raft_consensus.cc:729] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abaf18ee0752429da7894a08cec63fe0, State: Initialized, Role: FOLLOWER
I20260927 02:42:15.871156 2012 consensus_queue.cc:260] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } }
I20260927 02:42:15.871411 1987 heartbeater.cc:503] Master 127.0.98.62:35795 was elected leader, sending a full tablet report...
I20260927 02:42:15.871397 2012 ts_tablet_manager.cc:1428] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:15.872529 2013 raft_consensus.cc:348] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } }
I20260927 02:42:15.872664 2013 raft_consensus.cc:374] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:15.872685 2013 raft_consensus.cc:729] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3273e35dae6245c68789c144902a594d, State: Initialized, Role: FOLLOWER
I20260927 02:42:15.872774 2013 consensus_queue.cc:260] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } }
I20260927 02:42:15.872989 2013 ts_tablet_manager.cc:1428] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:15.873113 1707 heartbeater.cc:503] Master 127.0.98.62:35795 was elected leader, sending a full tablet report...
W20260927 02:42:15.925143 1708 tablet.cc:2369] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:16.044101 2017 raft_consensus.cc:482] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:16.044231 2017 raft_consensus.cc:504] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } }
I20260927 02:42:16.044577 2017 leader_election.cc:288] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c6150b5383c4a51b88482c579c118a3 (127.0.98.2:42985), 3273e35dae6245c68789c144902a594d (127.0.98.1:45897)
I20260927 02:42:16.047415 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.044874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45455 (local address 127.0.98.1:45897)
0927 02:42:16.044913 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.044915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.044927 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:42:16.044959 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.044959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.044970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.045045 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:16.045317 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.045798 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.046744 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.047105 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.047166 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.047176 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.047304 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.047340 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:42:16.047415 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.044843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59295 (local address 127.0.98.2:42985)
0927 02:42:16.044903 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.044904 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.045072 (+ 168us) server_negotiation.cc:407] Connection header received
0927 02:42:16.045103 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.045104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.045117 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.045186 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:16.045326 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.045798 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.046748 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.047099 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.047156 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.047166 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.047315 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.047350 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:42:16.047439 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.044791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45897 (local address 127.0.98.3:45455)
0927 02:42:16.044851 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:16.044863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:16.044906 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:16.045094 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:16.045095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:16.045097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:16.045233 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.045236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.045872 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.045873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.046653 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.046656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.047180 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.047181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.047243 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.047245 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:42:16.047288 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:42:16.047330 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":28}
I20260927 02:42:16.047456 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.044799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42985 (local address 127.0.98.3:59295)
0927 02:42:16.044977 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:16.044987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:16.045045 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:16.045192 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:16.045193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:16.045194 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:16.045255 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.045257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.045873 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.045874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.046655 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.046657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.047174 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.047175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.047248 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.047251 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:16.047297 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:42:16.047334 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":18,"thread_start_us":104,"threads_started":1}
I20260927 02:42:16.047755 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b254ac1b78ef4048bc61c910cab76322" candidate_uuid: "abaf18ee0752429da7894a08cec63fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3273e35dae6245c68789c144902a594d" is_pre_election: true
I20260927 02:42:16.047899 1661 raft_consensus.cc:2454] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abaf18ee0752429da7894a08cec63fe0 in term 0.
I20260927 02:42:16.047850 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b254ac1b78ef4048bc61c910cab76322" candidate_uuid: "abaf18ee0752429da7894a08cec63fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6150b5383c4a51b88482c579c118a3" is_pre_election: true
I20260927 02:42:16.048007 1801 raft_consensus.cc:2454] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abaf18ee0752429da7894a08cec63fe0 in term 0.
I20260927 02:42:16.048110 1872 leader_election.cc:302] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3273e35dae6245c68789c144902a594d, abaf18ee0752429da7894a08cec63fe0; no voters:
I20260927 02:42:16.048221 2017 raft_consensus.cc:2781] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:16.048266 2017 raft_consensus.cc:482] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:16.048287 2017 raft_consensus.cc:3037] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:16.049217 2017 raft_consensus.cc:504] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } }
I20260927 02:42:16.049331 2017 leader_election.cc:288] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [CANDIDATE]: Term 1 election: Requested vote from peers 2c6150b5383c4a51b88482c579c118a3 (127.0.98.2:42985), 3273e35dae6245c68789c144902a594d (127.0.98.1:45897)
I20260927 02:42:16.049520 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b254ac1b78ef4048bc61c910cab76322" candidate_uuid: "abaf18ee0752429da7894a08cec63fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6150b5383c4a51b88482c579c118a3"
I20260927 02:42:16.049533 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b254ac1b78ef4048bc61c910cab76322" candidate_uuid: "abaf18ee0752429da7894a08cec63fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3273e35dae6245c68789c144902a594d"
I20260927 02:42:16.049587 1801 raft_consensus.cc:3037] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:16.049621 1661 raft_consensus.cc:3037] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:16.050259 1801 raft_consensus.cc:2454] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abaf18ee0752429da7894a08cec63fe0 in term 1.
I20260927 02:42:16.050354 1661 raft_consensus.cc:2454] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abaf18ee0752429da7894a08cec63fe0 in term 1.
I20260927 02:42:16.050467 1873 leader_election.cc:302] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6150b5383c4a51b88482c579c118a3, abaf18ee0752429da7894a08cec63fe0; no voters:
I20260927 02:42:16.050592 2017 raft_consensus.cc:2781] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:16.050746 2017 raft_consensus.cc:686] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 1 LEADER]: Becoming Leader. State: Replica: abaf18ee0752429da7894a08cec63fe0, State: Running, Role: LEADER
I20260927 02:42:16.050836 2017 consensus_queue.cc:237] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } }
I20260927 02:42:16.051669 1503 catalog_manager.cc:5495] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abaf18ee0752429da7894a08cec63fe0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "abaf18ee0752429da7894a08cec63fe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } health_report { overall_health: UNKNOWN } } }
W20260927 02:42:16.066207 1988 tablet.cc:2369] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:42:16.076555 1848 tablet.cc:2369] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:16.091416 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.088977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.0.98.3:40103)
0927 02:42:16.089028 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.089030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.089038 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:16.089110 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.089111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.089129 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.089176 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:16.089340 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.089804 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.090622 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.090820 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.090874 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.091070 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:16.091163 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:16.091237 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.091338 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.091355 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:42:16.093268 1801 raft_consensus.cc:1261] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [term 1 FOLLOWER]: Refusing update from remote peer abaf18ee0752429da7894a08cec63fe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:16.093293 1661 raft_consensus.cc:1261] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Refusing update from remote peer abaf18ee0752429da7894a08cec63fe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:16.093621 2020 consensus_queue.cc:1035] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:16.093747 2017 consensus_queue.cc:1035] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:16.096510 2024 mvcc.cc:205] Tried to move back new op lower bound from 7333793530235392000 to 7333793530064867328. Current Snapshot: MvccSnapshot[applied={T|T < 7333793530235392000 or (T in {7333793530235392000})}]
I20260927 02:42:16.100008 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:16.100134 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:16.101119 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b: Sending DeleteTablet for 3 replicas of tablet b254ac1b78ef4048bc61c910cab76322
I20260927 02:42:16.101502 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet b254ac1b78ef4048bc61c910cab76322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43036
I20260927 02:42:16.101536 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet b254ac1b78ef4048bc61c910cab76322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53042
I20260927 02:42:16.101856 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet b254ac1b78ef4048bc61c910cab76322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35700
I20260927 02:42:16.101917 2034 tablet_replica.cc:330] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: stopping tablet replica
I20260927 02:42:16.101886 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:16.102033 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:16.102077 2034 raft_consensus.cc:2229] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:16.102097 2033 tablet_replica.cc:330] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: stopping tablet replica
I20260927 02:42:16.102210 2033 raft_consensus.cc:2229] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:16.102312 2033 raft_consensus.cc:2258] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:16.103049 2033 ts_tablet_manager.cc:1905] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:16.103260 2034 raft_consensus.cc:2258] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:16.103314 1641 tablet_service.cc:1467] Processing CreateTablet for tablet fb2b3af31ed04d40847c69c0083711c2 (DEFAULT_TABLE table=test-table [id=ba8fc906be014299ae133fc3ccc1a68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:16.103350 1781 tablet_service.cc:1467] Processing CreateTablet for tablet fb2b3af31ed04d40847c69c0083711c2 (DEFAULT_TABLE table=test-table [id=ba8fc906be014299ae133fc3ccc1a68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:16.103468 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2b3af31ed04d40847c69c0083711c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:16.103519 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2b3af31ed04d40847c69c0083711c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:16.103726 2034 ts_tablet_manager.cc:1905] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:16.104903 2013 tablet_bootstrap.cc:492] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: Bootstrap starting.
I20260927 02:42:16.104969 2010 tablet_bootstrap.cc:492] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: Bootstrap starting.
I20260927 02:42:16.105399 2010 tablet_bootstrap.cc:654] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:16.105844 2013 tablet_bootstrap.cc:654] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:16.106101 1921 tablet_service.cc:1467] Processing CreateTablet for tablet fb2b3af31ed04d40847c69c0083711c2 (DEFAULT_TABLE table=test-table [id=ba8fc906be014299ae133fc3ccc1a68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:16.106189 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2b3af31ed04d40847c69c0083711c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:16.106577 2034 ts_tablet_manager.cc:1918] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:16.106642 2034 log.cc:1196] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/b254ac1b78ef4048bc61c910cab76322
I20260927 02:42:16.106693 2010 tablet_bootstrap.cc:492] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: No bootstrap required, opened a new log
I20260927 02:42:16.106729 2010 ts_tablet_manager.cc:1397] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:16.106849 2010 raft_consensus.cc:348] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.106896 2010 raft_consensus.cc:374] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:16.106910 2010 raft_consensus.cc:729] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6150b5383c4a51b88482c579c118a3, State: Initialized, Role: FOLLOWER
I20260927 02:42:16.106874 2034 ts_tablet_manager.cc:1939] T b254ac1b78ef4048bc61c910cab76322 P abaf18ee0752429da7894a08cec63fe0: Deleting consensus metadata
I20260927 02:42:16.106951 2010 consensus_queue.cc:260] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.107017 2033 ts_tablet_manager.cc:1918] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:16.107054 2035 tablet_replica.cc:330] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: stopping tablet replica
I20260927 02:42:16.107079 2033 log.cc:1196] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/b254ac1b78ef4048bc61c910cab76322
I20260927 02:42:16.107178 2035 raft_consensus.cc:2229] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:16.107306 2033 ts_tablet_manager.cc:1939] T b254ac1b78ef4048bc61c910cab76322 P 3273e35dae6245c68789c144902a594d: Deleting consensus metadata
I20260927 02:42:16.107326 2012 tablet_bootstrap.cc:492] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: Bootstrap starting.
I20260927 02:42:16.107715 2012 tablet_bootstrap.cc:654] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:16.107697 1488 catalog_manager.cc:4841] TS 3273e35dae6245c68789c144902a594d (127.0.98.1:45897): tablet b254ac1b78ef4048bc61c910cab76322 (table test-table [id=ac616b69df9347a89d7161f9be910e2a]) successfully deleted
I20260927 02:42:16.107060 2010 ts_tablet_manager.cc:1428] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:16.108224 1489 catalog_manager.cc:4841] TS abaf18ee0752429da7894a08cec63fe0 (127.0.98.3:40103): tablet b254ac1b78ef4048bc61c910cab76322 (table test-table [id=ac616b69df9347a89d7161f9be910e2a]) successfully deleted
I20260927 02:42:16.108510 2035 raft_consensus.cc:2258] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:16.108644 2012 tablet_bootstrap.cc:492] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: No bootstrap required, opened a new log
I20260927 02:42:16.108711 2012 ts_tablet_manager.cc:1397] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:16.108955 2013 tablet_bootstrap.cc:492] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: No bootstrap required, opened a new log
I20260927 02:42:16.108927 2012 raft_consensus.cc:348] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.109004 2012 raft_consensus.cc:374] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:16.109007 2013 ts_tablet_manager.cc:1397] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 02:42:16.109025 2012 raft_consensus.cc:729] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abaf18ee0752429da7894a08cec63fe0, State: Initialized, Role: FOLLOWER
I20260927 02:42:16.109079 2012 consensus_queue.cc:260] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.109174 2012 ts_tablet_manager.cc:1428] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:16.109159 2013 raft_consensus.cc:348] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.109236 2013 raft_consensus.cc:374] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:16.109261 2013 raft_consensus.cc:729] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3273e35dae6245c68789c144902a594d, State: Initialized, Role: FOLLOWER
I20260927 02:42:16.109332 2013 consensus_queue.cc:260] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.109455 2013 ts_tablet_manager.cc:1428] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:16.109490 2035 ts_tablet_manager.cc:1905] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:16.111207 2035 ts_tablet_manager.cc:1918] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:16.111280 2035 log.cc:1196] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/b254ac1b78ef4048bc61c910cab76322
I20260927 02:42:16.111801 2035 ts_tablet_manager.cc:1939] T b254ac1b78ef4048bc61c910cab76322 P 2c6150b5383c4a51b88482c579c118a3: Deleting consensus metadata
I20260927 02:42:16.112262 1489 catalog_manager.cc:4841] TS 2c6150b5383c4a51b88482c579c118a3 (127.0.98.2:42985): tablet b254ac1b78ef4048bc61c910cab76322 (table test-table [id=ac616b69df9347a89d7161f9be910e2a]) successfully deleted
I20260927 02:42:16.163497 2016 raft_consensus.cc:482] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:16.163595 2016 raft_consensus.cc:504] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.163897 2016 leader_election.cc:288] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3273e35dae6245c68789c144902a594d (127.0.98.1:45897), abaf18ee0752429da7894a08cec63fe0 (127.0.98.3:40103)
I20260927 02:42:16.167400 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.164210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49583 (local address 127.0.98.3:40103)
0927 02:42:16.164287 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.164289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.164322 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:42:16.164352 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.164353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.164363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.164441 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:16.164610 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.165344 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.166726 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.167090 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.167151 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.167162 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.167303 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.167337 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:42:16.167438 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.164079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40103 (local address 127.0.98.2:49583)
0927 02:42:16.164222 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:16.164233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:16.164287 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:16.164454 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:16.164455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:16.164457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:16.164542 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.164544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.165419 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.165420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.166609 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.166613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.167163 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.167164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.167236 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.167241 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:16.167285 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:42:16.167328 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":7,"thread_start_us":77,"threads_started":1}
I20260927 02:42:16.167706 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2b3af31ed04d40847c69c0083711c2" candidate_uuid: "2c6150b5383c4a51b88482c579c118a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abaf18ee0752429da7894a08cec63fe0" is_pre_election: true
I20260927 02:42:16.167819 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.164262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42479 (local address 127.0.98.1:45897)
0927 02:42:16.164299 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.164301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.164403 (+ 102us) server_negotiation.cc:407] Connection header received
0927 02:42:16.164486 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.164486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.164495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.164524 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:16.164695 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.165146 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.166706 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.167242 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.167310 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.167320 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.167706 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.167749 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:42:16.167842 1941 raft_consensus.cc:2454] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6150b5383c4a51b88482c579c118a3 in term 0.
I20260927 02:42:16.168022 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.164091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45897 (local address 127.0.98.2:42479)
0927 02:42:16.164334 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:16.164342 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:16.164412 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:16.164578 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:16.164579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:16.164581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:16.164630 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.164631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.165278 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.165279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.166623 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.166625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.167368 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.167369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.167608 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.167613 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:16.167902 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 02:42:16.167945 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":26,"thread_start_us":74,"threads_started":1}
I20260927 02:42:16.168032 1733 leader_election.cc:302] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6150b5383c4a51b88482c579c118a3, abaf18ee0752429da7894a08cec63fe0; no voters:
I20260927 02:42:16.168185 2016 raft_consensus.cc:2781] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:16.168222 2016 raft_consensus.cc:482] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:16.168241 2016 raft_consensus.cc:3037] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:16.168303 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2b3af31ed04d40847c69c0083711c2" candidate_uuid: "2c6150b5383c4a51b88482c579c118a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3273e35dae6245c68789c144902a594d" is_pre_election: true
I20260927 02:42:16.168409 1661 raft_consensus.cc:2454] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6150b5383c4a51b88482c579c118a3 in term 0.
I20260927 02:42:16.169000 2016 raft_consensus.cc:504] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.169116 2016 leader_election.cc:288] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [CANDIDATE]: Term 1 election: Requested vote from peers 3273e35dae6245c68789c144902a594d (127.0.98.1:45897), abaf18ee0752429da7894a08cec63fe0 (127.0.98.3:40103)
I20260927 02:42:16.169270 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2b3af31ed04d40847c69c0083711c2" candidate_uuid: "2c6150b5383c4a51b88482c579c118a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abaf18ee0752429da7894a08cec63fe0"
I20260927 02:42:16.169287 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2b3af31ed04d40847c69c0083711c2" candidate_uuid: "2c6150b5383c4a51b88482c579c118a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3273e35dae6245c68789c144902a594d"
I20260927 02:42:16.169344 1941 raft_consensus.cc:3037] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:16.169344 1661 raft_consensus.cc:3037] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:16.170058 1661 raft_consensus.cc:2454] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6150b5383c4a51b88482c579c118a3 in term 1.
I20260927 02:42:16.170058 1941 raft_consensus.cc:2454] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6150b5383c4a51b88482c579c118a3 in term 1.
I20260927 02:42:16.170238 1732 leader_election.cc:302] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6150b5383c4a51b88482c579c118a3, 3273e35dae6245c68789c144902a594d; no voters:
I20260927 02:42:16.170329 2016 raft_consensus.cc:2781] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:16.170460 2016 raft_consensus.cc:686] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6150b5383c4a51b88482c579c118a3, State: Running, Role: LEADER
I20260927 02:42:16.170523 2016 consensus_queue.cc:237] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.171188 1504 catalog_manager.cc:5495] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c6150b5383c4a51b88482c579c118a3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2c6150b5383c4a51b88482c579c118a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:42:16 dist-test-slave-44kr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:16.212926 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.205164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35795 (local address 127.0.0.1:45824)
0927 02:42:16.205396 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:16.205408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:16.205485 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:16.205616 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:16.205616 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:16.206003 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:16.206063 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.206066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.206741 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.206742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.207571 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.207574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.207944 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.207944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.208009 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.208593 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:16.208598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:16.210198 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:16.211865 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:16.211872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:16.211874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:16.212098 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:16.212297 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:16.212298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:16.212298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:16.212355 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:16.212633 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:16.212640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:16.212686 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 02:42:16.212748 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:42:16.212845 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":136,"threads_started":1}
I20260927 02:42:16.213027 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.205277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.0.98.62:35795)
0927 02:42:16.205349 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.205351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.205475 (+ 124us) server_negotiation.cc:407] Connection header received
0927 02:42:16.205533 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.205534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.205542 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.205591 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:16.206154 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.206658 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.207665 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.207871 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.207926 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.210278 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:16.210279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:16.210279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:16.210287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:16.211772 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:16.212172 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:16.212173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:16.212174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:16.212236 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:16.212432 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:16.212433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:16.212434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:16.212566 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:16.212629 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.212782 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.212907 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":46}
I20260927 02:42:16.215065 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45824:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:42:16.215297 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:16.216931 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1066f1bf2f4f34bdfdeeddd8834089 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d43d4c07c7354085a62830cd62207072]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:42:16.216931 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1066f1bf2f4f34bdfdeeddd8834089 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d43d4c07c7354085a62830cd62207072]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:42:16.216985 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1066f1bf2f4f34bdfdeeddd8834089 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d43d4c07c7354085a62830cd62207072]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:42:16.217109 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1066f1bf2f4f34bdfdeeddd8834089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:16.217119 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1066f1bf2f4f34bdfdeeddd8834089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:16.217127 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1066f1bf2f4f34bdfdeeddd8834089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:16.218576 2012 tablet_bootstrap.cc:492] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: Bootstrap starting.
I20260927 02:42:16.218578 2013 tablet_bootstrap.cc:492] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: Bootstrap starting.
I20260927 02:42:16.218888 2012 tablet_bootstrap.cc:654] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:16.218950 2013 tablet_bootstrap.cc:654] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:16.219156 2010 tablet_bootstrap.cc:492] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: Bootstrap starting.
I20260927 02:42:16.219514 2010 tablet_bootstrap.cc:654] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:16.220168 2010 tablet_bootstrap.cc:492] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: No bootstrap required, opened a new log
I20260927 02:42:16.220222 2010 ts_tablet_manager.cc:1397] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:16.220362 2010 raft_consensus.cc:348] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.220463 2010 raft_consensus.cc:374] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:16.220489 2010 raft_consensus.cc:729] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6150b5383c4a51b88482c579c118a3, State: Initialized, Role: FOLLOWER
I20260927 02:42:16.220539 2010 consensus_queue.cc:260] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.220628 2010 ts_tablet_manager.cc:1428] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:16.220784 2012 tablet_bootstrap.cc:492] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: No bootstrap required, opened a new log
I20260927 02:42:16.220778 2016 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.220834 2016 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:16.220849 2012 ts_tablet_manager.cc:1397] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:16.220958 2012 raft_consensus.cc:348] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.221024 2012 raft_consensus.cc:374] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:16.221048 2012 raft_consensus.cc:729] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abaf18ee0752429da7894a08cec63fe0, State: Initialized, Role: FOLLOWER
I20260927 02:42:16.221088 2012 consensus_queue.cc:260] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.221185 2012 ts_tablet_manager.cc:1428] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:16.221390 2029 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.221464 2029 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:16.222312 2013 tablet_bootstrap.cc:492] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: No bootstrap required, opened a new log
I20260927 02:42:16.222368 2013 ts_tablet_manager.cc:1397] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 02:42:16.222506 2013 raft_consensus.cc:348] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.222568 2013 raft_consensus.cc:374] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:16.222586 2013 raft_consensus.cc:729] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3273e35dae6245c68789c144902a594d, State: Initialized, Role: FOLLOWER
I20260927 02:42:16.222627 2013 consensus_queue.cc:260] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.222754 2013 ts_tablet_manager.cc:1428] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:16.222882 2018 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.222939 2018 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:16.284806 2016 raft_consensus.cc:482] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:16.284893 2016 raft_consensus.cc:504] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.285050 2016 leader_election.cc:288] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3273e35dae6245c68789c144902a594d (127.0.98.1:45897), abaf18ee0752429da7894a08cec63fe0 (127.0.98.3:40103)
I20260927 02:42:16.285261 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1066f1bf2f4f34bdfdeeddd8834089" candidate_uuid: "2c6150b5383c4a51b88482c579c118a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abaf18ee0752429da7894a08cec63fe0" is_pre_election: true
I20260927 02:42:16.285261 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1066f1bf2f4f34bdfdeeddd8834089" candidate_uuid: "2c6150b5383c4a51b88482c579c118a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3273e35dae6245c68789c144902a594d" is_pre_election: true
I20260927 02:42:16.285370 1661 raft_consensus.cc:2454] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6150b5383c4a51b88482c579c118a3 in term 0.
I20260927 02:42:16.285370 1941 raft_consensus.cc:2454] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6150b5383c4a51b88482c579c118a3 in term 0.
I20260927 02:42:16.285509 1733 leader_election.cc:302] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6150b5383c4a51b88482c579c118a3, abaf18ee0752429da7894a08cec63fe0; no voters:
I20260927 02:42:16.285595 2016 raft_consensus.cc:2781] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:16.285647 2016 raft_consensus.cc:482] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:16.285665 2016 raft_consensus.cc:3037] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:16.286422 2016 raft_consensus.cc:504] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.286530 2016 leader_election.cc:288] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [CANDIDATE]: Term 1 election: Requested vote from peers 3273e35dae6245c68789c144902a594d (127.0.98.1:45897), abaf18ee0752429da7894a08cec63fe0 (127.0.98.3:40103)
I20260927 02:42:16.286705 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1066f1bf2f4f34bdfdeeddd8834089" candidate_uuid: "2c6150b5383c4a51b88482c579c118a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3273e35dae6245c68789c144902a594d"
I20260927 02:42:16.286705 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1066f1bf2f4f34bdfdeeddd8834089" candidate_uuid: "2c6150b5383c4a51b88482c579c118a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abaf18ee0752429da7894a08cec63fe0"
I20260927 02:42:16.286793 1941 raft_consensus.cc:3037] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:16.286793 1661 raft_consensus.cc:3037] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:16.287533 1661 raft_consensus.cc:2454] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6150b5383c4a51b88482c579c118a3 in term 1.
I20260927 02:42:16.287533 1941 raft_consensus.cc:2454] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6150b5383c4a51b88482c579c118a3 in term 1.
I20260927 02:42:16.287700 1732 leader_election.cc:302] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6150b5383c4a51b88482c579c118a3, 3273e35dae6245c68789c144902a594d; no voters:
I20260927 02:42:16.287806 2016 raft_consensus.cc:2781] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:16.287871 2016 raft_consensus.cc:686] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6150b5383c4a51b88482c579c118a3, State: Running, Role: LEADER
I20260927 02:42:16.287912 2016 consensus_queue.cc:237] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } }
I20260927 02:42:16.288208 2041 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: TxnStatusTablet state changed. Reason: New leader 2c6150b5383c4a51b88482c579c118a3. Latest consensus state: current_term: 1 leader_uuid: "2c6150b5383c4a51b88482c579c118a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.288277 2041 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:42:16.288470 2053 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:42:16.288570 2053 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:42:16.288537 1503 catalog_manager.cc:5495] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c6150b5383c4a51b88482c579c118a3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2c6150b5383c4a51b88482c579c118a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:16.296445 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.294102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.0.98.2:42985)
0927 02:42:16.294144 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.294146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.294154 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:16.294240 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.294241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.294260 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.294313 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:16.294466 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.294971 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.295710 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.295906 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.295959 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.296124 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:16.296222 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:16.296280 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.296326 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.296347 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:42:16.296459 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.293995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42985 (local address 127.0.0.1:35708)
0927 02:42:16.294088 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:16.294102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:16.294159 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:16.294344 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:16.294345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:16.294346 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:16.294409 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.294411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.295048 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.295049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.295638 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.295640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.295964 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.295965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.296031 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.296047 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:16.296275 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:16.296281 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:42:16.296324 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:42:16.296377 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:42:16.297501 1661 raft_consensus.cc:1261] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Refusing update from remote peer 2c6150b5383c4a51b88482c579c118a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:16.297601 1941 raft_consensus.cc:1261] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0 [term 1 FOLLOWER]: Refusing update from remote peer 2c6150b5383c4a51b88482c579c118a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:16.297799 2041 consensus_queue.cc:1035] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:16.297936 2016 consensus_queue.cc:1035] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:16.298581 2029 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: TxnStatusTablet state changed. Reason: New leader 2c6150b5383c4a51b88482c579c118a3. Latest consensus state: current_term: 1 leader_uuid: "2c6150b5383c4a51b88482c579c118a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.298664 2029 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:16.299391 2041 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c6150b5383c4a51b88482c579c118a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.299468 2041 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:42:16.299436 2018 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: TxnStatusTablet state changed. Reason: New leader 2c6150b5383c4a51b88482c579c118a3. Latest consensus state: current_term: 1 leader_uuid: "2c6150b5383c4a51b88482c579c118a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.299525 2018 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:16.300045 2018 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c6150b5383c4a51b88482c579c118a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.300134 2018 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 3273e35dae6245c68789c144902a594d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:16.300112 2016 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c6150b5383c4a51b88482c579c118a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.300190 2016 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P 2c6150b5383c4a51b88482c579c118a3: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:42:16.301072 2029 tablet_replica.cc:439] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c6150b5383c4a51b88482c579c118a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 } } peers { permanent_uuid: "2c6150b5383c4a51b88482c579c118a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 } } }
I20260927 02:42:16.301155 2029 tablet_replica.cc:442] T 0f1066f1bf2f4f34bdfdeeddd8834089 P abaf18ee0752429da7894a08cec63fe0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:16.301312 2023 mvcc.cc:205] Tried to move back new op lower bound from 7333793531072585728 to 7333793531035533312. Current Snapshot: MvccSnapshot[applied={T|T < 7333793531072585728 or (T in {7333793531072585728})}]
I20260927 02:42:16.304049 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.301537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.0.98.2:42985)
0927 02:42:16.301654 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.301655 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.301660 (+ 5us) server_negotiation.cc:407] Connection header received
0927 02:42:16.301675 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.301676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.301694 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.301735 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:16.301885 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.302394 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.303458 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.303642 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.303698 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.303829 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:16.303885 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:16.303929 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.303980 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.303998 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 02:42:16.308633 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.306435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40357 (local address 127.0.98.2:42985)
0927 02:42:16.306468 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.306469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.306589 (+ 120us) server_negotiation.cc:407] Connection header received
0927 02:42:16.306641 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.306642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.306650 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.306687 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:16.306817 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.307276 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.307987 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.308171 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.308221 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.308357 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:16.308412 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:16.308453 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.308503 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.308542 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":13}
I20260927 02:42:16.308693 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.306390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42985 (local address 127.0.98.2:40357)
0927 02:42:16.306533 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:16.306542 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:16.306592 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:16.306687 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:16.306688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:16.306690 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:16.306766 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.306768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.307342 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.307343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.307921 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:16.307923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.308229 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.308229 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:16.308284 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.308297 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:16.308452 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:16.308453 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:42:16.308499 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:42:16.308564 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":15,"thread_start_us":92,"threads_started":1}
I20260927 02:42:16.311362 1941 raft_consensus.cc:1261] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [term 1 FOLLOWER]: Refusing update from remote peer 2c6150b5383c4a51b88482c579c118a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:16.311367 1661 raft_consensus.cc:1261] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Refusing update from remote peer 2c6150b5383c4a51b88482c579c118a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:16.311594 2061 consensus_queue.cc:1035] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abaf18ee0752429da7894a08cec63fe0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:16.311702 2041 consensus_queue.cc:1035] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3273e35dae6245c68789c144902a594d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:16.314234 2025 mvcc.cc:205] Tried to move back new op lower bound from 7333793531129966592 to 7333793530554904576. Current Snapshot: MvccSnapshot[applied={T|T < 7333793531129966592}]
I20260927 02:42:16.325752 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:16.325848 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:16.326400 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1648330d718646c09c1df5a2f718191b: Sending DeleteTablet for 3 replicas of tablet fb2b3af31ed04d40847c69c0083711c2
I20260927 02:42:16.326673 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2b3af31ed04d40847c69c0083711c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43036
I20260927 02:42:16.326725 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2b3af31ed04d40847c69c0083711c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53042
I20260927 02:42:16.326778 2033 tablet_replica.cc:330] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: stopping tablet replica
I20260927 02:42:16.326807 2034 tablet_replica.cc:330] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: stopping tablet replica
I20260927 02:42:16.326844 2034 raft_consensus.cc:2229] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:16.326843 2033 raft_consensus.cc:2229] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:16.326874 2034 raft_consensus.cc:2258] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:16.326875 2033 raft_consensus.cc:2258] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:16.327041 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet fb2b3af31ed04d40847c69c0083711c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35700
I20260927 02:42:16.327126 2035 tablet_replica.cc:330] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: stopping tablet replica
I20260927 02:42:16.327172 2035 raft_consensus.cc:2229] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:16.327267 2035 raft_consensus.cc:2258] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:16.327579 2033 ts_tablet_manager.cc:1905] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:16.327579 2035 ts_tablet_manager.cc:1905] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:16.327877 2034 ts_tablet_manager.cc:1905] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:16.329169 2033 ts_tablet_manager.cc:1918] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:16.329239 2033 log.cc:1196] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/fb2b3af31ed04d40847c69c0083711c2
I20260927 02:42:16.329241 2035 ts_tablet_manager.cc:1918] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:16.329288 2035 log.cc:1196] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/fb2b3af31ed04d40847c69c0083711c2
I20260927 02:42:16.329290 2034 ts_tablet_manager.cc:1918] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:16.329332 2034 log.cc:1196] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/fb2b3af31ed04d40847c69c0083711c2
I20260927 02:42:16.329460 2033 ts_tablet_manager.cc:1939] T fb2b3af31ed04d40847c69c0083711c2 P 3273e35dae6245c68789c144902a594d: Deleting consensus metadata
I20260927 02:42:16.329487 2035 ts_tablet_manager.cc:1939] T fb2b3af31ed04d40847c69c0083711c2 P 2c6150b5383c4a51b88482c579c118a3: Deleting consensus metadata
I20260927 02:42:16.329505 2034 ts_tablet_manager.cc:1939] T fb2b3af31ed04d40847c69c0083711c2 P abaf18ee0752429da7894a08cec63fe0: Deleting consensus metadata
I20260927 02:42:16.329855 1489 catalog_manager.cc:4841] TS 2c6150b5383c4a51b88482c579c118a3 (127.0.98.2:42985): tablet fb2b3af31ed04d40847c69c0083711c2 (table test-table [id=ba8fc906be014299ae133fc3ccc1a68a]) successfully deleted
I20260927 02:42:16.330129 1488 catalog_manager.cc:4841] TS 3273e35dae6245c68789c144902a594d (127.0.98.1:45897): tablet fb2b3af31ed04d40847c69c0083711c2 (table test-table [id=ba8fc906be014299ae133fc3ccc1a68a]) successfully deleted
I20260927 02:42:16.330135 1489 catalog_manager.cc:4841] TS abaf18ee0752429da7894a08cec63fe0 (127.0.98.3:40103): tablet fb2b3af31ed04d40847c69c0083711c2 (table test-table [id=ba8fc906be014299ae133fc3ccc1a68a]) successfully deleted
Sep 27 02:42:16 dist-test-slave-44kr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:16.344693 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.335590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.0.98.1:45897)
0927 02:42:16.335653 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.335655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.335800 (+ 145us) server_negotiation.cc:407] Connection header received
0927 02:42:16.335866 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.335866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.335877 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.335923 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:16.336666 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.337224 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.338268 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.338453 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.338505 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.341645 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:16.341654 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:16.341655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:16.341662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:16.343384 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:16.343777 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:16.343780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:16.343781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:16.343850 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:16.344107 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:16.344108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:16.344109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:16.344265 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:16.344417 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.344527 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.344604 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 02:42:16.344923 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43052
I20260927 02:42:16.352584 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:16.345847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.0.98.62:35795)
0927 02:42:16.345900 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:16.345902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:16.346075 (+ 173us) server_negotiation.cc:407] Connection header received
0927 02:42:16.346123 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:16.346124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:16.346135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:16.346201 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:16.346660 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.347436 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.348170 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:16.348516 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:16.348585 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:16.349807 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:16.349808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:16.349809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:16.349817 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:16.351405 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:16.351740 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:16.351741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:16.351742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:16.351802 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:16.352012 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:16.352012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:16.352013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:16.352143 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:16.352235 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:16.352371 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:42:16.352483 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 02:42:16.352818 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45830
I20260927 02:42:16.353495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 1576
I20260927 02:42:16.357775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 1716
I20260927 02:42:16.362571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 1856
I20260927 02:42:16.367298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 1472
2026-09-27T02:42:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2619 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:16 dist-test-slave-44kr krb5kdc[2071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:16 dist-test-slave-44kr krb5kdc[2071](info): set up 2 sockets
Sep 27 02:42:16 dist-test-slave-44kr krb5kdc[2071](info): commencing operation
krb5kdc: starting...
W20260927 02:42:17.324004 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:17 dist-test-slave-44kr krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:17.439670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40341
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:17.519886 2084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:17.520116 2084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:17.520139 2084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:17.521360 2084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:17.521409 2084 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:42:17.521421 2084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:17.521433 2084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:17.521445 2084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:17.521456 2084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:17.522998 2084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40341
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:17.523249 2084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:17.523488 2084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:17.528426 2090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:17.528451 2084 server_base.cc:1034] running on GCE node
W20260927 02:42:17.528419 2092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:17.528431 2089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:17.529145 2084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:17.529453 2084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:17.530629 2084 hybrid_clock.cc:648] HybridClock initialized: now 1790476937530627 us; error 43 us; skew 500 ppm
Sep 27 02:42:17 dist-test-slave-44kr krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:17.533185 2084 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:17.533603 2084 webserver.cc:460] Webserver started at http://127.0.98.62:42355/ using document root <none> and password file <none>
I20260927 02:42:17.533814 2084 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:17.533946 2084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:17.534096 2084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:17.535233 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "2dd11ad2ca1840a787face04fa33ebe3"
format_stamp: "Formatted at 2026-09-27 02:42:17 on dist-test-slave-44kr"
server_key: "1e30d948173e1696f2a6223da4792abc"
server_key_iv: "484c2cde54ddbaa5a3dcf53897b57fcd"
server_key_version: "encryptionkey@0"
I20260927 02:42:17.535561 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "2dd11ad2ca1840a787face04fa33ebe3"
format_stamp: "Formatted at 2026-09-27 02:42:17 on dist-test-slave-44kr"
server_key: "1e30d948173e1696f2a6223da4792abc"
server_key_iv: "484c2cde54ddbaa5a3dcf53897b57fcd"
server_key_version: "encryptionkey@0"
I20260927 02:42:17.536850 2084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:17.537506 2099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:17.537655 2084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:42:17.537746 2084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "2dd11ad2ca1840a787face04fa33ebe3"
format_stamp: "Formatted at 2026-09-27 02:42:17 on dist-test-slave-44kr"
server_key: "1e30d948173e1696f2a6223da4792abc"
server_key_iv: "484c2cde54ddbaa5a3dcf53897b57fcd"
server_key_version: "encryptionkey@0"
I20260927 02:42:17.537806 2084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:17.564339 2084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:17.566332 2084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:17.566473 2084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:17.570899 2084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40341
I20260927 02:42:17.570967 2161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40341 every 8 connection(s)
I20260927 02:42:17.571267 2084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:17.571918 2162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:17.574211 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: Bootstrap starting.
I20260927 02:42:17.574816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 2084
I20260927 02:42:17.574841 2162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:17.574906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:17.575080 392 external_mini_cluster.cc:1428] Setting key 341af3623d143cbcd88c08178e530096
I20260927 02:42:17.575136 2162 log.cc:824] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:17.575891 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: No bootstrap required, opened a new log
I20260927 02:42:17.577490 2162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd11ad2ca1840a787face04fa33ebe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40341 } }
I20260927 02:42:17.577638 2162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:17.577684 2162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd11ad2ca1840a787face04fa33ebe3, State: Initialized, Role: FOLLOWER
I20260927 02:42:17.577813 2162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd11ad2ca1840a787face04fa33ebe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40341 } }
I20260927 02:42:17.577886 2162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:17.577926 2162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:17.577972 2162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:17.578514 2162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd11ad2ca1840a787face04fa33ebe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40341 } }
I20260927 02:42:17.578605 2162 leader_election.cc:302] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dd11ad2ca1840a787face04fa33ebe3; no voters:
I20260927 02:42:17.578773 2162 leader_election.cc:288] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:17.578810 2167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:17.578920 2167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd11ad2ca1840a787face04fa33ebe3, State: Running, Role: LEADER
I20260927 02:42:17.579002 2167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd11ad2ca1840a787face04fa33ebe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40341 } }
I20260927 02:42:17.579035 2162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:17.580273 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dd11ad2ca1840a787face04fa33ebe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd11ad2ca1840a787face04fa33ebe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40341 } } }
I20260927 02:42:17.580358 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:17.580360 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd11ad2ca1840a787face04fa33ebe3. Latest consensus state: current_term: 1 leader_uuid: "2dd11ad2ca1840a787face04fa33ebe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd11ad2ca1840a787face04fa33ebe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40341 } } }
I20260927 02:42:17.580477 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3 [sys.catalog]: This master's current role is: LEADER
W20260927 02:42:17.581194 2183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:17.581250 2183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:17.581568 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:17.581760 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:42:17 dist-test-slave-44kr krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:17.584332 2179 catalog_manager.cc:1347] Generated new cluster ID: 597de84bda1a46b983b70b41e6339fc7
I20260927 02:42:17.584404 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:17.585935 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:17.575564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.0.98.62:40341)
0927 02:42:17.575760 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:17.575767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:17.575793 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:42:17.576249 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:17.576257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:17.576309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:17.576458 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:17.577043 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.578319 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.579247 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.579837 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.579946 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:17.582603 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:17.582607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:17.582607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:17.582631 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:17.584217 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.584623 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.584627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.584627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.584745 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.584957 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.584957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.584958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.585311 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:17.585482 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:17.585556 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:42:17.585718 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260927 02:42:17.590692 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:17.591197 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:17.597486 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: Generated new TSK 0
I20260927 02:42:17.597643 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:17.627737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40341
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:17.744114 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:17.744336 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:17.744364 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:17.745682 2188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:17.745733 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:17.745764 2188 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:42:17.745791 2188 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:42:17.745823 2188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:17.747383 2188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:17.747695 2188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:17.747934 2188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:17.753381 2194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:17.753486 2188 server_base.cc:1034] running on GCE node
W20260927 02:42:17.753398 2196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:17.753409 2193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:17.753854 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:17.754113 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:17.755273 2188 hybrid_clock.cc:648] HybridClock initialized: now 1790476937755237 us; error 52 us; skew 500 ppm
Sep 27 02:42:17 dist-test-slave-44kr krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:17.758040 2188 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:17.758554 2188 webserver.cc:460] Webserver started at http://127.0.98.1:41717/ using document root <none> and password file <none>
I20260927 02:42:17.758785 2188 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:17.758831 2188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:17.758955 2188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:17.759979 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "7babeff2a1b54da19f65be2c0d82e666"
format_stamp: "Formatted at 2026-09-27 02:42:17 on dist-test-slave-44kr"
server_key: "7335f148dc817a2ae6d2efa7c2a5a577"
server_key_iv: "27af7f286d9b2122de62d4d015cb7e98"
server_key_version: "encryptionkey@0"
I20260927 02:42:17.760320 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7babeff2a1b54da19f65be2c0d82e666"
format_stamp: "Formatted at 2026-09-27 02:42:17 on dist-test-slave-44kr"
server_key: "7335f148dc817a2ae6d2efa7c2a5a577"
server_key_iv: "27af7f286d9b2122de62d4d015cb7e98"
server_key_version: "encryptionkey@0"
I20260927 02:42:17.761631 2188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:17.762435 2203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:17.762598 2188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:17.762682 2188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "7babeff2a1b54da19f65be2c0d82e666"
format_stamp: "Formatted at 2026-09-27 02:42:17 on dist-test-slave-44kr"
server_key: "7335f148dc817a2ae6d2efa7c2a5a577"
server_key_iv: "27af7f286d9b2122de62d4d015cb7e98"
server_key_version: "encryptionkey@0"
I20260927 02:42:17.762746 2188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:17.774333 2188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:17.776537 2188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:17.776659 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:17.777137 2188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:17.777177 2188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:17.777204 2188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:17.777215 2188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:17.784112 2188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41781
I20260927 02:42:17.784178 2318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41781 every 8 connection(s)
I20260927 02:42:17.784577 2188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:42:17 dist-test-slave-44kr krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:17.788046 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:17.777707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42227 (local address 127.0.98.62:40341)
0927 02:42:17.777758 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:17.777760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:17.777982 (+ 222us) server_negotiation.cc:407] Connection header received
0927 02:42:17.778513 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:17.778515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:17.778527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:17.778588 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:17.779602 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.780461 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.781999 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.782312 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.782371 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:17.785176 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:17.785178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:17.785178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:17.785187 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:17.786553 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.786975 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.786976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.786977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.787031 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.787354 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.787355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.787355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.787476 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:17.787528 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:17.787891 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 02:42:17.787977 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:42:17.788151 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:17.777678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40341 (local address 127.0.98.1:42227)
0927 02:42:17.777856 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:17.777893 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:17.778409 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:17.778712 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:17.778718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:17.779115 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:17.779476 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:17.779484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.780563 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.780564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:17.781882 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:17.781886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.782395 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.782396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:17.782548 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:17.783195 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:17.783206 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:17.785047 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:17.786663 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:17.786670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:17.786672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:17.786908 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:17.787112 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:17.787112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:17.787113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:17.787296 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:17.787559 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:17.787565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:17.787735 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 02:42:17.787832 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:42:17.787974 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260927 02:42:17.788703 2319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40341
I20260927 02:42:17.788801 2319 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:17.789016 2319 heartbeater.cc:511] Master 127.0.98.62:40341 requested a full tablet report, sending...
I20260927 02:42:17.789650 2115 ts_manager.cc:194] Registered new tserver with Master: 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781)
I20260927 02:42:17.790354 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42227
I20260927 02:42:17.793592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 2188
I20260927 02:42:17.793690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:17.793866 392 external_mini_cluster.cc:1428] Setting key 591fdb62f6ab5000ccf8c58de88f8f5d
I20260927 02:42:17.796036 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:17.790542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60475 (local address 127.0.98.62:40341)
0927 02:42:17.790606 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:17.790608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:17.790616 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:17.790636 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:17.790636 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:17.790646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:17.790698 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:17.791027 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.791518 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.792209 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.792451 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.792515 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:17.793553 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:17.793554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:17.793554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:17.793561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:17.795003 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.795337 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.795338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.795338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.795396 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.795556 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.795557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.795558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.795666 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:17.795719 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:17.795859 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 02:42:17.795939 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:42:17.796051 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:17.789728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40341 (local address 127.0.98.1:60475)
0927 02:42:17.789859 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:17.789872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:17.789936 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:17.790729 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:17.790730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:17.790901 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:17.790957 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:17.790959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.791620 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.791621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:17.792136 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:17.792138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.792510 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.792511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:17.792582 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:17.793075 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:17.793080 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:17.793462 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:17.795083 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:17.795084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:17.795084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:17.795272 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:17.795449 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:17.795450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:17.795450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:17.795508 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:17.795724 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:17.795725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:17.795772 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 02:42:17.795831 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:42:17.795941 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:17.823196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40341
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:17.905130 2328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:17.905346 2328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:17.905370 2328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:17.906491 2328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:17.906530 2328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:17.906549 2328 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:42:17.906559 2328 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:42:17.906575 2328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:17.908027 2328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:17.908236 2328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:17.908452 2328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:17.913571 2334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:17.913614 2336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:17.913622 2333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:17.913883 2328 server_base.cc:1034] running on GCE node
I20260927 02:42:17.914119 2328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:17.914336 2328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:17.915503 2328 hybrid_clock.cc:648] HybridClock initialized: now 1790476937915485 us; error 28 us; skew 500 ppm
Sep 27 02:42:17 dist-test-slave-44kr krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:17.918243 2328 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:17.918691 2328 webserver.cc:460] Webserver started at http://127.0.98.2:44279/ using document root <none> and password file <none>
I20260927 02:42:17.918911 2328 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:17.918962 2328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:17.919097 2328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:17.920317 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "d33e8a8f637041c2bafd065c4a27e643"
format_stamp: "Formatted at 2026-09-27 02:42:17 on dist-test-slave-44kr"
server_key: "9933bfcdff07e179dd27496d650fee2d"
server_key_iv: "a3ba05f6184a84495b8fc6c810578e68"
server_key_version: "encryptionkey@0"
I20260927 02:42:17.920732 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d33e8a8f637041c2bafd065c4a27e643"
format_stamp: "Formatted at 2026-09-27 02:42:17 on dist-test-slave-44kr"
server_key: "9933bfcdff07e179dd27496d650fee2d"
server_key_iv: "a3ba05f6184a84495b8fc6c810578e68"
server_key_version: "encryptionkey@0"
I20260927 02:42:17.922101 2328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:17.922912 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:17.923154 2328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:17.923244 2328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "d33e8a8f637041c2bafd065c4a27e643"
format_stamp: "Formatted at 2026-09-27 02:42:17 on dist-test-slave-44kr"
server_key: "9933bfcdff07e179dd27496d650fee2d"
server_key_iv: "a3ba05f6184a84495b8fc6c810578e68"
server_key_version: "encryptionkey@0"
I20260927 02:42:17.923312 2328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:17.942049 2328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:17.944693 2328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:17.944872 2328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:17.945456 2328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:17.945510 2328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:17.945534 2328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:17.945546 2328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:17.952049 2328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34993
I20260927 02:42:17.952114 2458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34993 every 8 connection(s)
Sep 27 02:42:17 dist-test-slave-44kr krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:17.952436 2328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:17.955523 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:17.945997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45815 (local address 127.0.98.62:40341)
0927 02:42:17.946075 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:17.946077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:17.946346 (+ 269us) server_negotiation.cc:407] Connection header received
0927 02:42:17.946956 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:17.946957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:17.946970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:17.947000 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:17.948030 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.948815 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.950108 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.950389 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.950449 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:17.952932 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:17.952934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:17.952934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:17.952941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:17.954205 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.954626 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.954627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.954628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.954693 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.954959 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.954960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.954960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.955071 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:17.955169 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:17.955398 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 02:42:17.955462 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:42:17.955730 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:17.945960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40341 (local address 127.0.98.2:45815)
0927 02:42:17.946214 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:17.946263 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:17.946832 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:17.947216 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:17.947220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:17.947608 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:17.947901 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:17.947908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.948921 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.948923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:17.950025 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:17.950028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.950464 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.950464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:17.950599 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:17.951173 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:17.951186 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:17.952853 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:17.954292 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:17.954297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:17.954299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:17.954520 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:17.954786 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:17.954787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:17.954787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:17.954896 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:17.955170 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:17.955174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:17.955293 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 02:42:17.955377 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:42:17.955510 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260927 02:42:17.956184 2459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40341
I20260927 02:42:17.956264 2459 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:17.956462 2459 heartbeater.cc:511] Master 127.0.98.62:40341 requested a full tablet report, sending...
I20260927 02:42:17.956825 2115 ts_manager.cc:194] Registered new tserver with Master: d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993)
I20260927 02:42:17.957311 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45815
I20260927 02:42:17.958330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 2328
I20260927 02:42:17.958410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:17.958582 392 external_mini_cluster.cc:1428] Setting key b31995e7d52dcb53f70d63474f25c407
I20260927 02:42:17.962749 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:17.957244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45241 (local address 127.0.98.62:40341)
0927 02:42:17.957285 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:17.957287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:17.957339 (+ 52us) server_negotiation.cc:407] Connection header received
0927 02:42:17.957403 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:17.957404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:17.957414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:17.957466 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:17.957792 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.958368 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.959241 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.959441 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.959492 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:17.960470 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:17.960471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:17.960471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:17.960477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:17.961668 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.962033 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.962033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.962034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.962074 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:17.962263 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:17.962264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:17.962264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:17.962418 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:17.962471 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:17.962626 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 02:42:17.962684 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:17.962811 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:17.957148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40341 (local address 127.0.98.2:45241)
0927 02:42:17.957283 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:17.957294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:17.957359 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:17.957480 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:17.957481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:17.957670 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:17.957729 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:17.957731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.958463 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.958464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:17.959182 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:17.959185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:17.959497 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:17.959498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:17.959571 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:17.959986 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:17.959989 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:17.960340 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:17.961760 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:17.961761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:17.961762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:17.961960 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:17.962131 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:17.962132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:17.962132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:17.962194 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:17.962480 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:17.962481 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:17.962529 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 02:42:17.962599 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:42:17.962706 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:17.984097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40341
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:18.062932 2468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:18.063146 2468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:18.063176 2468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:18.064330 2468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:18.064376 2468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:18.064445 2468 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:42:18.064460 2468 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:42:18.064483 2468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:18.065977 2468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:18.066220 2468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:18.066437 2468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:42:18.071517 2468 server_base.cc:1034] running on GCE node
W20260927 02:42:18.071501 2476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:18.071501 2474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:18.071521 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:18.071906 2468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:18.072221 2468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:18.073392 2468 hybrid_clock.cc:648] HybridClock initialized: now 1790476938073357 us; error 61 us; skew 500 ppm
Sep 27 02:42:18 dist-test-slave-44kr krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:18.075908 2468 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:18.076423 2468 webserver.cc:460] Webserver started at http://127.0.98.3:44059/ using document root <none> and password file <none>
I20260927 02:42:18.076646 2468 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:18.076690 2468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:18.076798 2468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:18.077813 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "0fcbb82e91744384a6bc65f71bed4828"
format_stamp: "Formatted at 2026-09-27 02:42:18 on dist-test-slave-44kr"
server_key: "ca69ec896da59643fb78b01c76ded5d2"
server_key_iv: "13952169bf9a8fe1e346171217beb6bd"
server_key_version: "encryptionkey@0"
I20260927 02:42:18.078138 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0fcbb82e91744384a6bc65f71bed4828"
format_stamp: "Formatted at 2026-09-27 02:42:18 on dist-test-slave-44kr"
server_key: "ca69ec896da59643fb78b01c76ded5d2"
server_key_iv: "13952169bf9a8fe1e346171217beb6bd"
server_key_version: "encryptionkey@0"
I20260927 02:42:18.079520 2468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:18.080436 2483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:18.080610 2468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:18.080700 2468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "0fcbb82e91744384a6bc65f71bed4828"
format_stamp: "Formatted at 2026-09-27 02:42:18 on dist-test-slave-44kr"
server_key: "ca69ec896da59643fb78b01c76ded5d2"
server_key_iv: "13952169bf9a8fe1e346171217beb6bd"
server_key_version: "encryptionkey@0"
I20260927 02:42:18.080771 2468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:18.105537 2468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:18.107919 2468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:18.108078 2468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:18.108632 2468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:18.108672 2468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:18.108701 2468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:18.108718 2468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:18.115239 2468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43397
I20260927 02:42:18.115296 2598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43397 every 8 connection(s)
Sep 27 02:42:18 dist-test-slave-44kr krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:18.115741 2468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:18.119129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 2468
I20260927 02:42:18.119174 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.109259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42943 (local address 127.0.98.62:40341)
0927 02:42:18.109327 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.109329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.109598 (+ 269us) server_negotiation.cc:407] Connection header received
0927 02:42:18.110117 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.110118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.110132 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.110200 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:18.111076 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.111865 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.113401 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.113688 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.113762 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.116364 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:18.116366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:18.116366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:18.116373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:18.117708 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:18.118139 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:18.118141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:18.118141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:18.118211 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:18.118426 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:18.118426 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:18.118427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:18.118542 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:18.118593 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.119009 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.119109 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:42:18.119227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:18.119321 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.109231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40341 (local address 127.0.98.3:42943)
0927 02:42:18.109407 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.109452 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.110009 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.110296 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.110301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.110645 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:18.110946 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.110953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.111990 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.111991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.113313 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.113317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.113786 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.113787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.113896 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.114595 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:18.114605 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:18.116263 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:18.117803 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:18.117810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:18.117812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:18.118047 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:18.118275 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:18.118276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:18.118276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:18.118377 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:18.118658 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:18.118664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:18.118823 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.118920 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.119084 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260927 02:42:18.119421 392 external_mini_cluster.cc:1428] Setting key e043c6a3478fbc69d1529a365cf4fff8
I20260927 02:42:18.119848 2599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40341
I20260927 02:42:18.119936 2599 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:18.120126 2599 heartbeater.cc:511] Master 127.0.98.62:40341 requested a full tablet report, sending...
I20260927 02:42:18.120911 2115 ts_manager.cc:194] Registered new tserver with Master: 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397)
I20260927 02:42:18.121330 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:18.121397 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42943
I20260927 02:42:18.126770 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.120858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40341 (local address 127.0.98.3:51557)
0927 02:42:18.120999 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.121009 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.121098 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.121203 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.121204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.121391 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:18.121451 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.121454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.122078 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.122079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.122796 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.122798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.123256 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.123257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.123308 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.123725 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:18.123729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:18.124056 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:18.125568 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:18.125569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:18.125569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:18.125833 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:18.126185 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:18.126186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:18.126186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:18.126252 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:18.126506 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:18.126507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:18.126544 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.126593 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.126676 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260927 02:42:18.126808 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.120882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51557 (local address 127.0.98.62:40341)
0927 02:42:18.120915 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.120916 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.121054 (+ 138us) server_negotiation.cc:407] Connection header received
0927 02:42:18.121145 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.121146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.121156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.121199 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:18.121512 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.121974 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.122891 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.123179 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.123248 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.124192 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:18.124193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:18.124193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:18.124199 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:18.125467 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:18.125947 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:18.125950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:18.125950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:18.126112 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:18.126315 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:18.126316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:18.126317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:18.126447 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:18.126518 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.126641 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.126723 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Sep 27 02:42:18 dist-test-slave-44kr krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:42:18 dist-test-slave-44kr krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:18.144330 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.137366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.0.98.62:40341)
0927 02:42:18.137445 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.137447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.137454 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:42:18.137477 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.137478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.137486 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.137583 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:18.138169 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.138655 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.139396 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.139576 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.139624 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.141919 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:18.141920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:18.141921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:18.141928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:18.143219 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:18.143515 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:18.143516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:18.143516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:18.143567 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:18.143807 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:18.143809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:18.143809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:18.143930 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:18.144020 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.144188 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.144259 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:42:18.146020 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:18.147686 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:18.154237 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.150062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41781 (local address 127.0.0.1:37562)
0927 02:42:18.150344 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.150384 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.150455 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.150709 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.150714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.150717 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:18.150788 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.150790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.151774 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.151775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.153203 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.153206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.153928 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.153929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.154012 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.154026 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.154091 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.154148 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260927 02:42:18.154246 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.150156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.0.98.1:41781)
0927 02:42:18.150309 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.150314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.150452 (+ 138us) server_negotiation.cc:407] Connection header received
0927 02:42:18.150536 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.150539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.150555 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.150628 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:18.150894 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.151701 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.153294 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.153854 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.153918 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.153980 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.154111 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.154147 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260927 02:42:18.154446 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.150070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34993 (local address 127.0.0.1:57462)
0927 02:42:18.150557 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.150565 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.150602 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.150957 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.150957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.150959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:18.150991 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.150992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.152253 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.152254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.153372 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.153373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.154243 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.154244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.154296 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.154299 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.154342 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.154383 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":31,"thread_start_us":93,"threads_started":1}
I20260927 02:42:18.154450 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.150314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.0.98.3:43397)
0927 02:42:18.150459 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.150464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.150766 (+ 302us) server_negotiation.cc:407] Connection header received
0927 02:42:18.150808 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.150810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.150824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.150894 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:18.151046 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.152167 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.153256 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.153952 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.154021 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.154075 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.154263 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.154334 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260927 02:42:18.154747 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.150482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.0.98.2:34993)
0927 02:42:18.150621 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.150625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.150634 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:18.150677 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.150678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.150690 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.150768 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:18.151067 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.152167 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.153443 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.154109 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.154167 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.154215 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.154618 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.154658 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260927 02:42:18.155282 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 009224ec09dd4a37a145b6c8b2a6c1bf (DEFAULT_TABLE table=test-table [id=96cb9093704a45a29e6831a7ff7a4c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:18.155498 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 009224ec09dd4a37a145b6c8b2a6c1bf (DEFAULT_TABLE table=test-table [id=96cb9093704a45a29e6831a7ff7a4c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:18.155625 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009224ec09dd4a37a145b6c8b2a6c1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:18.155821 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009224ec09dd4a37a145b6c8b2a6c1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:18.158771 2623 tablet_bootstrap.cc:492] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: Bootstrap starting.
I20260927 02:42:18.158928 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.150125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43397 (local address 127.0.0.1:39408)
0927 02:42:18.150692 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.150700 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.150747 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.150894 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.150895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.150897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:18.150955 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.150957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.152266 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.152267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.153175 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.153178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.154112 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.154113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.154173 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.154177 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.158782 (+ 4605us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.158852 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":76,"thread_start_us":63,"threads_started":1}
I20260927 02:42:18.158998 2622 tablet_bootstrap.cc:492] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: Bootstrap starting.
I20260927 02:42:18.159500 2623 tablet_bootstrap.cc:654] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:18.159586 2622 tablet_bootstrap.cc:654] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:18.159797 2623 log.cc:824] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:18.159709 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 009224ec09dd4a37a145b6c8b2a6c1bf (DEFAULT_TABLE table=test-table [id=96cb9093704a45a29e6831a7ff7a4c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:18.159907 2622 log.cc:824] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:18.160054 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009224ec09dd4a37a145b6c8b2a6c1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:18.160511 2623 tablet_bootstrap.cc:492] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: No bootstrap required, opened a new log
I20260927 02:42:18.160581 2623 ts_tablet_manager.cc:1397] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:18.160622 2622 tablet_bootstrap.cc:492] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: No bootstrap required, opened a new log
I20260927 02:42:18.160696 2622 ts_tablet_manager.cc:1397] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:18.162151 2623 raft_consensus.cc:348] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.162285 2623 raft_consensus.cc:374] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:18.162317 2623 raft_consensus.cc:729] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7babeff2a1b54da19f65be2c0d82e666, State: Initialized, Role: FOLLOWER
I20260927 02:42:18.162433 2623 consensus_queue.cc:260] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.162559 2622 raft_consensus.cc:348] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.162717 2623 ts_tablet_manager.cc:1428] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:18.162729 2622 raft_consensus.cc:374] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:18.162765 2622 raft_consensus.cc:729] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33e8a8f637041c2bafd065c4a27e643, State: Initialized, Role: FOLLOWER
I20260927 02:42:18.162724 2319 heartbeater.cc:503] Master 127.0.98.62:40341 was elected leader, sending a full tablet report...
I20260927 02:42:18.162885 2622 consensus_queue.cc:260] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.163153 2622 ts_tablet_manager.cc:1428] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:18.163200 2459 heartbeater.cc:503] Master 127.0.98.62:40341 was elected leader, sending a full tablet report...
I20260927 02:42:18.163355 2628 tablet_bootstrap.cc:492] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: Bootstrap starting.
I20260927 02:42:18.164239 2628 tablet_bootstrap.cc:654] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:18.164539 2628 log.cc:824] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:18.165191 2628 tablet_bootstrap.cc:492] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: No bootstrap required, opened a new log
I20260927 02:42:18.165252 2628 ts_tablet_manager.cc:1397] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:42:18.166638 2628 raft_consensus.cc:348] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.166769 2628 raft_consensus.cc:374] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:18.166788 2628 raft_consensus.cc:729] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcbb82e91744384a6bc65f71bed4828, State: Initialized, Role: FOLLOWER
I20260927 02:42:18.166873 2628 consensus_queue.cc:260] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.167069 2628 ts_tablet_manager.cc:1428] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:42:18.167124 2599 heartbeater.cc:503] Master 127.0.98.62:40341 was elected leader, sending a full tablet report...
W20260927 02:42:18.203020 2460 tablet.cc:2369] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:18.269513 2630 raft_consensus.cc:482] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:18.269649 2630 raft_consensus.cc:504] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.269995 2630 leader_election.cc:288] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781), d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993)
I20260927 02:42:18.273943 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.270186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41781 (local address 127.0.98.3:41417)
0927 02:42:18.270407 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.270417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.270475 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.270619 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.270620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.270623 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:18.270699 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.270701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.271536 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.271537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.272924 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.272927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.273661 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.273662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.273749 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.273754 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.273807 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.273844 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":56,"thread_start_us":145,"threads_started":1}
I20260927 02:42:18.273952 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.270292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41417 (local address 127.0.98.1:41781)
0927 02:42:18.270336 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.270338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.270502 (+ 164us) server_negotiation.cc:407] Connection header received
0927 02:42:18.270539 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.270540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.270553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.270616 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:18.270795 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.271436 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.273012 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.273582 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.273653 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.273665 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.273822 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.273868 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:18.273972 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.270293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46141 (local address 127.0.98.2:34993)
0927 02:42:18.270336 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.270338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.270348 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:18.270381 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.270382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.270394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.270467 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:18.270751 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.271473 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.273035 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.273593 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.273657 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.273668 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.273877 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.273917 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:18.274279 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.270186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34993 (local address 127.0.98.3:46141)
0927 02:42:18.270270 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.270281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.270328 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.270594 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.270595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.270598 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:18.270669 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.270671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.271542 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.271543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.272924 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.272927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.273694 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.273694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.273773 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.273775 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.273816 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.273844 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":23}
I20260927 02:42:18.274844 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009224ec09dd4a37a145b6c8b2a6c1bf" candidate_uuid: "0fcbb82e91744384a6bc65f71bed4828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7babeff2a1b54da19f65be2c0d82e666" is_pre_election: true
I20260927 02:42:18.274945 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009224ec09dd4a37a145b6c8b2a6c1bf" candidate_uuid: "0fcbb82e91744384a6bc65f71bed4828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33e8a8f637041c2bafd065c4a27e643" is_pre_election: true
I20260927 02:42:18.275015 2273 raft_consensus.cc:2454] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fcbb82e91744384a6bc65f71bed4828 in term 0.
I20260927 02:42:18.275074 2413 raft_consensus.cc:2454] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fcbb82e91744384a6bc65f71bed4828 in term 0.
I20260927 02:42:18.275259 2486 leader_election.cc:302] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcbb82e91744384a6bc65f71bed4828, 7babeff2a1b54da19f65be2c0d82e666; no voters:
I20260927 02:42:18.275388 2630 raft_consensus.cc:2781] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:18.275457 2630 raft_consensus.cc:482] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:18.275481 2630 raft_consensus.cc:3037] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:18.276412 2630 raft_consensus.cc:504] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.276666 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009224ec09dd4a37a145b6c8b2a6c1bf" candidate_uuid: "0fcbb82e91744384a6bc65f71bed4828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7babeff2a1b54da19f65be2c0d82e666"
I20260927 02:42:18.276746 2273 raft_consensus.cc:3037] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:18.276747 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009224ec09dd4a37a145b6c8b2a6c1bf" candidate_uuid: "0fcbb82e91744384a6bc65f71bed4828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33e8a8f637041c2bafd065c4a27e643"
I20260927 02:42:18.276816 2413 raft_consensus.cc:3037] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:18.277488 2273 raft_consensus.cc:2454] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fcbb82e91744384a6bc65f71bed4828 in term 1.
I20260927 02:42:18.277572 2413 raft_consensus.cc:2454] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fcbb82e91744384a6bc65f71bed4828 in term 1.
I20260927 02:42:18.277617 2486 leader_election.cc:302] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcbb82e91744384a6bc65f71bed4828, 7babeff2a1b54da19f65be2c0d82e666; no voters:
I20260927 02:42:18.277655 2630 leader_election.cc:288] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [CANDIDATE]: Term 1 election: Requested vote from peers 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781), d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993)
I20260927 02:42:18.277704 2630 raft_consensus.cc:2781] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:18.277750 2630 raft_consensus.cc:686] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 1 LEADER]: Becoming Leader. State: Replica: 0fcbb82e91744384a6bc65f71bed4828, State: Running, Role: LEADER
I20260927 02:42:18.277827 2630 consensus_queue.cc:237] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.278716 2115 catalog_manager.cc:5495] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0fcbb82e91744384a6bc65f71bed4828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } health_report { overall_health: HEALTHY } } }
W20260927 02:42:18.285336 2320 tablet.cc:2369] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:42:18 dist-test-slave-44kr krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:18.319967 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.313191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40341 (local address 127.0.0.1:48780)
0927 02:42:18.313380 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.313420 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.313497 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.313628 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.313629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.313938 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:18.313984 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.313986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.314715 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.314716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.315290 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.315293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.315619 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.315620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.315681 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.316183 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:18.316188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:18.317623 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:18.318958 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:18.318964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:18.318967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:18.319198 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:18.319374 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:18.319374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:18.319375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:18.319425 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:18.319669 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:18.319673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:18.319722 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.319788 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.319893 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260927 02:42:18.320050 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.313310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48780 (local address 127.0.98.62:40341)
0927 02:42:18.313387 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.313390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.313511 (+ 121us) server_negotiation.cc:407] Connection header received
0927 02:42:18.313541 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.313542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.313552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.313616 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:18.314105 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.314605 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.315374 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.315559 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.315618 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.317716 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:18.317717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:18.317718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:18.317724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:18.318889 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:18.319272 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:18.319273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:18.319273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:18.319324 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:18.319493 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:18.319494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:18.319495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:18.319613 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:18.319674 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.319822 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.319953 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":56}
I20260927 02:42:18.322116 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48780:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:42:18.322297 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:18.323642 2253 tablet_service.cc:1467] Processing CreateTablet for tablet c46aa558e8d54a17b53f5045254ed957 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b8704073dea4a9ea767fd23842da17a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:42:18.323698 2393 tablet_service.cc:1467] Processing CreateTablet for tablet c46aa558e8d54a17b53f5045254ed957 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b8704073dea4a9ea767fd23842da17a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:42:18.323827 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46aa558e8d54a17b53f5045254ed957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:18.323823 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46aa558e8d54a17b53f5045254ed957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:18.325037 2623 tablet_bootstrap.cc:492] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: Bootstrap starting.
I20260927 02:42:18.325047 2622 tablet_bootstrap.cc:492] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: Bootstrap starting.
I20260927 02:42:18.325315 2533 tablet_service.cc:1467] Processing CreateTablet for tablet c46aa558e8d54a17b53f5045254ed957 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b8704073dea4a9ea767fd23842da17a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:42:18.325408 2623 tablet_bootstrap.cc:654] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:18.325450 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46aa558e8d54a17b53f5045254ed957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:18.325717 2622 tablet_bootstrap.cc:654] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:18.325999 2623 tablet_bootstrap.cc:492] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: No bootstrap required, opened a new log
I20260927 02:42:18.326036 2623 ts_tablet_manager.cc:1397] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:18.326172 2623 raft_consensus.cc:348] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.326248 2623 raft_consensus.cc:374] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:18.326267 2623 raft_consensus.cc:729] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7babeff2a1b54da19f65be2c0d82e666, State: Initialized, Role: FOLLOWER
I20260927 02:42:18.326328 2622 tablet_bootstrap.cc:492] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: No bootstrap required, opened a new log
I20260927 02:42:18.326327 2623 consensus_queue.cc:260] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.326368 2622 ts_tablet_manager.cc:1397] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:18.326432 2623 ts_tablet_manager.cc:1428] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:18.326503 2622 raft_consensus.cc:348] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.326566 2622 raft_consensus.cc:374] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:18.326584 2622 raft_consensus.cc:729] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33e8a8f637041c2bafd065c4a27e643, State: Initialized, Role: FOLLOWER
I20260927 02:42:18.326622 2622 consensus_queue.cc:260] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.326696 2622 ts_tablet_manager.cc:1428] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:18.326766 2627 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.326802 2626 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.326836 2628 tablet_bootstrap.cc:492] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: Bootstrap starting.
I20260927 02:42:18.326845 2627 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:18.326867 2626 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:18.327201 2628 tablet_bootstrap.cc:654] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:18.327780 2628 tablet_bootstrap.cc:492] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: No bootstrap required, opened a new log
I20260927 02:42:18.327823 2628 ts_tablet_manager.cc:1397] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:18.327947 2628 raft_consensus.cc:348] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.328027 2628 raft_consensus.cc:374] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:18.328047 2628 raft_consensus.cc:729] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcbb82e91744384a6bc65f71bed4828, State: Initialized, Role: FOLLOWER
I20260927 02:42:18.328091 2628 consensus_queue.cc:260] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.328171 2628 ts_tablet_manager.cc:1428] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:18.328219 2630 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.328310 2630 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 02:42:18.366444 2600 tablet.cc:2369] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:18.521723 2626 raft_consensus.cc:482] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:18.521832 2626 raft_consensus.cc:504] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.522187 2626 leader_election.cc:288] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993), 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397)
I20260927 02:42:18.525112 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.522375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43397 (local address 127.0.98.1:49749)
0927 02:42:18.522515 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.522526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.522590 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.522742 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.522743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.522745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:18.522812 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.522814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.523439 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.523440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.524272 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.524274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.524867 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.524869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.524922 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.524926 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.524973 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.525012 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":18,"thread_start_us":68,"threads_started":1}
I20260927 02:42:18.525092 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.522460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49749 (local address 127.0.98.3:43397)
0927 02:42:18.522521 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.522523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.522593 (+ 70us) server_negotiation.cc:407] Connection header received
0927 02:42:18.522647 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.522648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.522660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.522706 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:18.522876 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.523358 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.524362 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.524760 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.524821 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.524831 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.524984 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.525019 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:42:18.525367 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.522461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33391 (local address 127.0.98.2:34993)
0927 02:42:18.522507 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.522509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.522645 (+ 136us) server_negotiation.cc:407] Connection header received
0927 02:42:18.522704 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.522705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.522714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.522763 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:18.522902 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.523390 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.524382 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.524757 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.524817 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.524828 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.525228 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.525277 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:42:18.525491 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46aa558e8d54a17b53f5045254ed957" candidate_uuid: "7babeff2a1b54da19f65be2c0d82e666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fcbb82e91744384a6bc65f71bed4828" is_pre_election: true
I20260927 02:42:18.525599 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.522375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34993 (local address 127.0.98.1:33391)
0927 02:42:18.522561 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.522570 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.522650 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.522772 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.522773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.522776 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:18.522848 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.522850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.523463 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.523463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.524297 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.524299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.525075 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.525076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.525134 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.525138 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.525486 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.525528 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":59,"thread_start_us":55,"threads_started":1}
I20260927 02:42:18.525628 2553 raft_consensus.cc:2454] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7babeff2a1b54da19f65be2c0d82e666 in term 0.
I20260927 02:42:18.525820 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46aa558e8d54a17b53f5045254ed957" candidate_uuid: "7babeff2a1b54da19f65be2c0d82e666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33e8a8f637041c2bafd065c4a27e643" is_pre_election: true
I20260927 02:42:18.525900 2413 raft_consensus.cc:2454] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7babeff2a1b54da19f65be2c0d82e666 in term 0.
I20260927 02:42:18.525921 2207 leader_election.cc:302] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcbb82e91744384a6bc65f71bed4828, 7babeff2a1b54da19f65be2c0d82e666; no voters:
I20260927 02:42:18.526057 2626 raft_consensus.cc:2781] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:18.526095 2626 raft_consensus.cc:482] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:18.526139 2626 raft_consensus.cc:3037] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:18.526784 2626 raft_consensus.cc:504] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.526909 2626 leader_election.cc:288] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [CANDIDATE]: Term 1 election: Requested vote from peers d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993), 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397)
I20260927 02:42:18.527055 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46aa558e8d54a17b53f5045254ed957" candidate_uuid: "7babeff2a1b54da19f65be2c0d82e666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33e8a8f637041c2bafd065c4a27e643"
I20260927 02:42:18.527060 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46aa558e8d54a17b53f5045254ed957" candidate_uuid: "7babeff2a1b54da19f65be2c0d82e666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fcbb82e91744384a6bc65f71bed4828"
I20260927 02:42:18.527128 2553 raft_consensus.cc:3037] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:18.527139 2413 raft_consensus.cc:3037] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:18.527778 2553 raft_consensus.cc:2454] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7babeff2a1b54da19f65be2c0d82e666 in term 1.
I20260927 02:42:18.527797 2413 raft_consensus.cc:2454] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7babeff2a1b54da19f65be2c0d82e666 in term 1.
I20260927 02:42:18.527940 2205 leader_election.cc:302] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7babeff2a1b54da19f65be2c0d82e666, d33e8a8f637041c2bafd065c4a27e643; no voters:
I20260927 02:42:18.528031 2626 raft_consensus.cc:2781] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:18.528153 2626 raft_consensus.cc:686] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [term 1 LEADER]: Becoming Leader. State: Replica: 7babeff2a1b54da19f65be2c0d82e666, State: Running, Role: LEADER
I20260927 02:42:18.528244 2626 consensus_queue.cc:237] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.528688 2645 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: TxnStatusTablet state changed. Reason: New leader 7babeff2a1b54da19f65be2c0d82e666. Latest consensus state: current_term: 1 leader_uuid: "7babeff2a1b54da19f65be2c0d82e666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.528767 2645 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:42:18.528937 2647 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:42:18.529035 2647 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:42:18.529076 2115 catalog_manager.cc:5495] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7babeff2a1b54da19f65be2c0d82e666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:18.555989 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.553211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41781 (local address 127.0.0.1:37576)
0927 02:42:18.553280 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.553292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.553339 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.553516 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.553516 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.553517 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:18.553561 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.553563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.554482 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.554483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.555077 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.555079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.555526 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.555526 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.555569 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.555584 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.555853 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.555859 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.555887 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.555932 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:42:18.555997 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.553285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37576 (local address 127.0.98.1:41781)
0927 02:42:18.553327 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.553330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.553339 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:18.553415 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.553415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.553436 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.553506 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:18.553611 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.554399 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.555150 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.555464 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.555531 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.555664 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.555795 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.555872 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.555893 (+ 21us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.555916 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:18.557951 2413 raft_consensus.cc:1261] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Refusing update from remote peer 7babeff2a1b54da19f65be2c0d82e666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:18.557982 2553 raft_consensus.cc:1261] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Refusing update from remote peer 7babeff2a1b54da19f65be2c0d82e666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:18.558259 2626 consensus_queue.cc:1035] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:18.558393 2645 consensus_queue.cc:1035] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:18.559518 2630 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: TxnStatusTablet state changed. Reason: New leader 7babeff2a1b54da19f65be2c0d82e666. Latest consensus state: current_term: 1 leader_uuid: "7babeff2a1b54da19f65be2c0d82e666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.559607 2630 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:18.559692 2627 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: TxnStatusTablet state changed. Reason: New leader 7babeff2a1b54da19f65be2c0d82e666. Latest consensus state: current_term: 1 leader_uuid: "7babeff2a1b54da19f65be2c0d82e666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.559780 2627 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:18.560999 2630 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7babeff2a1b54da19f65be2c0d82e666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.561084 2630 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P 0fcbb82e91744384a6bc65f71bed4828: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:18.561406 2627 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7babeff2a1b54da19f65be2c0d82e666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.561486 2627 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P d33e8a8f637041c2bafd065c4a27e643: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:18.561589 2626 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7babeff2a1b54da19f65be2c0d82e666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.561677 2626 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:42:18.561764 2626 tablet_replica.cc:439] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7babeff2a1b54da19f65be2c0d82e666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } } }
I20260927 02:42:18.561815 2626 tablet_replica.cc:442] T c46aa558e8d54a17b53f5045254ed957 P 7babeff2a1b54da19f65be2c0d82e666: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:42:18.562263 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333793540330852352 to 7333793540212314112. Current Snapshot: MvccSnapshot[applied={T|T < 7333793540330852352 or (T in {7333793540330852352})}]
I20260927 02:42:18.562430 2652 mvcc.cc:205] Tried to move back new op lower bound from 7333793540330852352 to 7333793540212314112. Current Snapshot: MvccSnapshot[applied={T|T < 7333793540330852352}]
I20260927 02:42:18.566136 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.563439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.0.98.3:43397)
0927 02:42:18.563479 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.563481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.563491 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:18.563522 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.563523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.563539 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.563627 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:18.563722 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.564499 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.565280 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.565601 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.565676 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.565825 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.565949 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.566023 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.566039 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.566060 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:42:18.571483 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.568882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40377 (local address 127.0.98.1:41781)
0927 02:42:18.568922 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.568923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.568931 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:18.568979 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.568979 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.568989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.569048 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:18.569152 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.569866 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.570658 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.570974 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.571040 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.571258 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.571292 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.571336 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.571392 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.571411 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:42:18.571502 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.568822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41781 (local address 127.0.98.3:40377)
0927 02:42:18.568891 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.568903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.568935 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.569051 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.569052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.569052 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:18.569101 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.569103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.569956 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.569957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.570555 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.570557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.571060 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.571061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.571129 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.571145 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.571344 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.571347 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.571390 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.571432 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:42:18.574357 2413 raft_consensus.cc:1261] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Refusing update from remote peer 0fcbb82e91744384a6bc65f71bed4828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:18.574393 2273 raft_consensus.cc:1261] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Refusing update from remote peer 0fcbb82e91744384a6bc65f71bed4828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:18.574630 2632 consensus_queue.cc:1035] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:18.574781 2630 consensus_queue.cc:1035] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:18.583609 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.580703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43397 (local address 127.0.98.1:38019)
0927 02:42:18.580843 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.580854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.580913 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.581029 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.581030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.581032 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:18.581091 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.581093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.581966 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.581967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.582837 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.582840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.583212 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.583213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.583281 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.583294 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.583441 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.583443 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.583488 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.583528 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260927 02:42:18.583585 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.580752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38019 (local address 127.0.98.3:43397)
0927 02:42:18.580794 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.580797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.580912 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:42:18.580960 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.580961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.580972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.581029 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:18.581150 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.581890 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.582932 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.583151 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.583205 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.583358 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.583393 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.583435 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.583492 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.583510 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:18.592664 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:18.592792 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:18.593556 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: Sending DeleteTablet for 3 replicas of tablet 009224ec09dd4a37a145b6c8b2a6c1bf
I20260927 02:42:18.593847 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 009224ec09dd4a37a145b6c8b2a6c1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39408
I20260927 02:42:18.593909 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 009224ec09dd4a37a145b6c8b2a6c1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37562
I20260927 02:42:18.594029 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:42:18.594116 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 009224ec09dd4a37a145b6c8b2a6c1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57462
W20260927 02:42:18.594183 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:18.594337 2670 tablet_replica.cc:330] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: stopping tablet replica
I20260927 02:42:18.594584 2670 raft_consensus.cc:2229] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:18.594825 2670 raft_consensus.cc:2258] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:18.595323 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc5d810747543f39f9e777231d6799b (DEFAULT_TABLE table=test-table [id=66b5dea40aeb4b1f9c1822bcee9f6a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:18.595453 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc5d810747543f39f9e777231d6799b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:18.595348 2670 ts_tablet_manager.cc:1905] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:18.595492 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc5d810747543f39f9e777231d6799b (DEFAULT_TABLE table=test-table [id=66b5dea40aeb4b1f9c1822bcee9f6a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:18.595774 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc5d810747543f39f9e777231d6799b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:18.596855 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc5d810747543f39f9e777231d6799b (DEFAULT_TABLE table=test-table [id=66b5dea40aeb4b1f9c1822bcee9f6a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:18.596975 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc5d810747543f39f9e777231d6799b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:18.597666 2670 ts_tablet_manager.cc:1918] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:18.597786 2670 log.cc:1196] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/009224ec09dd4a37a145b6c8b2a6c1bf
I20260927 02:42:18.597934 2622 tablet_bootstrap.cc:492] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: Bootstrap starting.
I20260927 02:42:18.597954 2623 tablet_bootstrap.cc:492] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: Bootstrap starting.
I20260927 02:42:18.598186 2670 ts_tablet_manager.cc:1939] T 009224ec09dd4a37a145b6c8b2a6c1bf P 0fcbb82e91744384a6bc65f71bed4828: Deleting consensus metadata
I20260927 02:42:18.598228 2623 tablet_bootstrap.cc:654] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:18.598313 2622 tablet_bootstrap.cc:654] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:18.598488 2628 tablet_bootstrap.cc:492] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: Bootstrap starting.
I20260927 02:42:18.598766 2103 catalog_manager.cc:4841] TS 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397): tablet 009224ec09dd4a37a145b6c8b2a6c1bf (table test-table [id=96cb9093704a45a29e6831a7ff7a4c61]) successfully deleted
I20260927 02:42:18.598845 2628 tablet_bootstrap.cc:654] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:18.599025 2622 tablet_bootstrap.cc:492] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: No bootstrap required, opened a new log
I20260927 02:42:18.599076 2622 ts_tablet_manager.cc:1397] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:18.599225 2622 raft_consensus.cc:348] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.599294 2622 raft_consensus.cc:374] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:18.599314 2622 raft_consensus.cc:729] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33e8a8f637041c2bafd065c4a27e643, State: Initialized, Role: FOLLOWER
I20260927 02:42:18.599351 2622 consensus_queue.cc:260] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.599431 2622 ts_tablet_manager.cc:1428] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:18.600504 2672 tablet_replica.cc:330] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: stopping tablet replica
I20260927 02:42:18.600653 2671 tablet_replica.cc:330] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: stopping tablet replica
I20260927 02:42:18.600883 2623 tablet_bootstrap.cc:492] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: No bootstrap required, opened a new log
I20260927 02:42:18.601045 2628 tablet_bootstrap.cc:492] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: No bootstrap required, opened a new log
I20260927 02:42:18.601048 2623 ts_tablet_manager.cc:1397] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:42:18.601081 2628 ts_tablet_manager.cc:1397] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 02:42:18.601189 2623 raft_consensus.cc:348] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.601253 2623 raft_consensus.cc:374] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:18.601225 2628 raft_consensus.cc:348] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.601274 2623 raft_consensus.cc:729] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7babeff2a1b54da19f65be2c0d82e666, State: Initialized, Role: FOLLOWER
I20260927 02:42:18.601284 2628 raft_consensus.cc:374] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:18.601305 2628 raft_consensus.cc:729] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcbb82e91744384a6bc65f71bed4828, State: Initialized, Role: FOLLOWER
I20260927 02:42:18.601315 2623 consensus_queue.cc:260] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.601344 2628 consensus_queue.cc:260] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.601398 2623 ts_tablet_manager.cc:1428] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:18.601421 2628 ts_tablet_manager.cc:1428] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:18.601784 2672 raft_consensus.cc:2229] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:18.602051 2671 raft_consensus.cc:2229] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:18.602154 2671 raft_consensus.cc:2258] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:18.602288 2672 raft_consensus.cc:2258] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:18.603091 2671 ts_tablet_manager.cc:1905] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:18.603366 2672 ts_tablet_manager.cc:1905] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:18.606561 2672 ts_tablet_manager.cc:1918] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:18.606748 2671 ts_tablet_manager.cc:1918] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:18.606859 2672 log.cc:1196] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/009224ec09dd4a37a145b6c8b2a6c1bf
I20260927 02:42:18.606949 2671 log.cc:1196] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/009224ec09dd4a37a145b6c8b2a6c1bf
I20260927 02:42:18.607242 2672 ts_tablet_manager.cc:1939] T 009224ec09dd4a37a145b6c8b2a6c1bf P d33e8a8f637041c2bafd065c4a27e643: Deleting consensus metadata
I20260927 02:42:18.607532 2671 ts_tablet_manager.cc:1939] T 009224ec09dd4a37a145b6c8b2a6c1bf P 7babeff2a1b54da19f65be2c0d82e666: Deleting consensus metadata
I20260927 02:42:18.608009 2102 catalog_manager.cc:4841] TS 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781): tablet 009224ec09dd4a37a145b6c8b2a6c1bf (table test-table [id=96cb9093704a45a29e6831a7ff7a4c61]) successfully deleted
I20260927 02:42:18.608332 2101 catalog_manager.cc:4841] TS d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993): tablet 009224ec09dd4a37a145b6c8b2a6c1bf (table test-table [id=96cb9093704a45a29e6831a7ff7a4c61]) successfully deleted
I20260927 02:42:18.805320 2627 raft_consensus.cc:482] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:18.805429 2627 raft_consensus.cc:504] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.805784 2627 leader_election.cc:288] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781), 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397)
I20260927 02:42:18.808696 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.806050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49729 (local address 127.0.98.1:41781)
0927 02:42:18.806098 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.806100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.806176 (+ 76us) server_negotiation.cc:407] Connection header received
0927 02:42:18.806241 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.806242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.806251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.806310 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:18.806467 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.806921 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.807999 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.808365 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.808446 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.808455 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.808594 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.808623 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:42:18.808707 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.806043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51439 (local address 127.0.98.3:43397)
0927 02:42:18.806107 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.806110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.806195 (+ 85us) server_negotiation.cc:407] Connection header received
0927 02:42:18.806250 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.806251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.806259 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.806306 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:18.806449 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.806949 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.807995 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.808347 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.808437 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.808450 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.808591 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.808624 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:42:18.808753 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.805972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41781 (local address 127.0.98.2:49729)
0927 02:42:18.806100 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.806111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.806183 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.806327 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.806328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.806330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:18.806408 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.806409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.806990 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.806991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.807907 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.807910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.808462 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.808462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.808532 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.808535 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.808582 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.808625 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":63,"thread_start_us":63,"threads_started":1}
I20260927 02:42:18.808902 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.805972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43397 (local address 127.0.98.2:51439)
0927 02:42:18.806106 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.806116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.806196 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.806310 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.806311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.806313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:18.806394 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.806395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.807025 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.807026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.807904 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.807907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.808452 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.808453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.808528 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.808531 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.808577 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.808622 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":51,"thread_start_us":67,"threads_started":1}
I20260927 02:42:18.808970 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc5d810747543f39f9e777231d6799b" candidate_uuid: "d33e8a8f637041c2bafd065c4a27e643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7babeff2a1b54da19f65be2c0d82e666" is_pre_election: true
I20260927 02:42:18.809070 2273 raft_consensus.cc:2454] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33e8a8f637041c2bafd065c4a27e643 in term 0.
I20260927 02:42:18.809180 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc5d810747543f39f9e777231d6799b" candidate_uuid: "d33e8a8f637041c2bafd065c4a27e643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fcbb82e91744384a6bc65f71bed4828" is_pre_election: true
I20260927 02:42:18.809227 2346 leader_election.cc:302] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7babeff2a1b54da19f65be2c0d82e666, d33e8a8f637041c2bafd065c4a27e643; no voters:
I20260927 02:42:18.809273 2553 raft_consensus.cc:2454] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33e8a8f637041c2bafd065c4a27e643 in term 0.
I20260927 02:42:18.809356 2627 raft_consensus.cc:2781] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:18.809397 2627 raft_consensus.cc:482] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:18.809422 2627 raft_consensus.cc:3037] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:18.810068 2627 raft_consensus.cc:504] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.810190 2627 leader_election.cc:288] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [CANDIDATE]: Term 1 election: Requested vote from peers 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781), 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397)
I20260927 02:42:18.810351 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc5d810747543f39f9e777231d6799b" candidate_uuid: "d33e8a8f637041c2bafd065c4a27e643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fcbb82e91744384a6bc65f71bed4828"
I20260927 02:42:18.810369 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc5d810747543f39f9e777231d6799b" candidate_uuid: "d33e8a8f637041c2bafd065c4a27e643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7babeff2a1b54da19f65be2c0d82e666"
I20260927 02:42:18.810423 2553 raft_consensus.cc:3037] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:18.810434 2273 raft_consensus.cc:3037] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:18.810961 2553 raft_consensus.cc:2454] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33e8a8f637041c2bafd065c4a27e643 in term 1.
I20260927 02:42:18.811084 2273 raft_consensus.cc:2454] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33e8a8f637041c2bafd065c4a27e643 in term 1.
I20260927 02:42:18.811144 2347 leader_election.cc:302] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcbb82e91744384a6bc65f71bed4828, d33e8a8f637041c2bafd065c4a27e643; no voters:
I20260927 02:42:18.811234 2627 raft_consensus.cc:2781] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:18.811375 2627 raft_consensus.cc:686] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 1 LEADER]: Becoming Leader. State: Replica: d33e8a8f637041c2bafd065c4a27e643, State: Running, Role: LEADER
I20260927 02:42:18.811440 2627 consensus_queue.cc:237] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:18.812211 2115 catalog_manager.cc:5495] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 reported cstate change: term changed from 0 to 1, leader changed from <none> to d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d33e8a8f637041c2bafd065c4a27e643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:18.831996 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.829779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.0.98.2:34993)
0927 02:42:18.829827 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.829829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.829839 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:18.829920 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.829921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.829941 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.830011 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:18.830175 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.830653 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.831350 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.831536 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.831594 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.831692 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.831792 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.831841 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.831923 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.831939 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:18.836588 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.834350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51259 (local address 127.0.98.1:41781)
0927 02:42:18.834401 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.834403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.834467 (+ 64us) server_negotiation.cc:407] Connection header received
0927 02:42:18.834535 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.834535 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.834545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.834585 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:18.834706 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.835155 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.835943 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.836119 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.836171 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.836334 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.836367 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.836451 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.836496 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.836513 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:42:18.836611 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.834266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41781 (local address 127.0.98.2:51259)
0927 02:42:18.834416 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.834427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.834481 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.834585 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.834585 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.834586 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:18.834651 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.834652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.835261 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.835262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.835854 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.835856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.836197 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.836198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.836256 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.836272 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.836453 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.836454 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.836494 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.836533 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260927 02:42:18.839063 2553 raft_consensus.cc:1261] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Refusing update from remote peer d33e8a8f637041c2bafd065c4a27e643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:18.839119 2273 raft_consensus.cc:1261] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Refusing update from remote peer d33e8a8f637041c2bafd065c4a27e643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:18.839324 2678 consensus_queue.cc:1035] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:18.839449 2627 consensus_queue.cc:1035] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:18.841069 2650 mvcc.cc:205] Tried to move back new op lower bound from 7333793541483180032 to 7333793541372190720. Current Snapshot: MvccSnapshot[applied={T|T < 7333793541483180032}]
I20260927 02:42:18.846036 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.843785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50061 (local address 127.0.98.2:34993)
0927 02:42:18.843824 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:18.843826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:18.843837 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:18.843890 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:18.843891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:18.843899 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:18.843943 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:18.844058 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.844651 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.845346 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.845544 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.845598 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.845785 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.845821 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.845883 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:18.845940 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 02:42:18.845962 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:42:18.846046 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:18.843756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34993 (local address 127.0.98.1:50061)
0927 02:42:18.843800 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:18.843811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:18.843850 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:18.843958 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:18.843959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:18.843961 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:18.843999 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.844001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.844717 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.844718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.845272 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:18.845275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:18.845662 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:18.845663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:18.845715 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:18.845722 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:18.845890 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:18.845892 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:42:18.845935 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:42:18.845980 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:42:18.846338 2652 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:42:20.364871 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:20.364976 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:20.366061 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: Sending DeleteTablet for 3 replicas of tablet 5bc5d810747543f39f9e777231d6799b
I20260927 02:42:20.366348 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc5d810747543f39f9e777231d6799b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39408
I20260927 02:42:20.366385 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc5d810747543f39f9e777231d6799b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57462
I20260927 02:42:20.366392 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc5d810747543f39f9e777231d6799b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37562
I20260927 02:42:20.366631 2710 tablet_replica.cc:330] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: stopping tablet replica
I20260927 02:42:20.366694 2710 raft_consensus.cc:2229] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:20.366737 2710 raft_consensus.cc:2258] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:20.366959 2710 ts_tablet_manager.cc:1905] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:20.368455 2710 ts_tablet_manager.cc:1918] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:42:20.368513 2710 log.cc:1196] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/5bc5d810747543f39f9e777231d6799b
I20260927 02:42:20.368748 2710 ts_tablet_manager.cc:1939] T 5bc5d810747543f39f9e777231d6799b P 7babeff2a1b54da19f65be2c0d82e666: Deleting consensus metadata
I20260927 02:42:20.369041 2102 catalog_manager.cc:4841] TS 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781): tablet 5bc5d810747543f39f9e777231d6799b (table test-table [id=66b5dea40aeb4b1f9c1822bcee9f6a6e]) successfully deleted
I20260927 02:42:20.370805 2708 tablet_replica.cc:330] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: stopping tablet replica
I20260927 02:42:20.370927 2708 raft_consensus.cc:2229] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:20.371008 2708 raft_consensus.cc:2258] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:20.371129 2709 tablet_replica.cc:330] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: stopping tablet replica
I20260927 02:42:20.371248 2709 raft_consensus.cc:2229] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:20.371292 2708 ts_tablet_manager.cc:1905] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:20.371389 2709 raft_consensus.cc:2258] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:20.371641 2709 ts_tablet_manager.cc:1905] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:20.372936 2708 ts_tablet_manager.cc:1918] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:42:20.372998 2708 log.cc:1196] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/5bc5d810747543f39f9e777231d6799b
I20260927 02:42:20.373199 2708 ts_tablet_manager.cc:1939] T 5bc5d810747543f39f9e777231d6799b P 0fcbb82e91744384a6bc65f71bed4828: Deleting consensus metadata
I20260927 02:42:20.373365 2709 ts_tablet_manager.cc:1918] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:42:20.373417 2709 log.cc:1196] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/5bc5d810747543f39f9e777231d6799b
I20260927 02:42:20.373591 2709 ts_tablet_manager.cc:1939] T 5bc5d810747543f39f9e777231d6799b P d33e8a8f637041c2bafd065c4a27e643: Deleting consensus metadata
I20260927 02:42:20.373638 2103 catalog_manager.cc:4841] TS 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397): tablet 5bc5d810747543f39f9e777231d6799b (table test-table [id=66b5dea40aeb4b1f9c1822bcee9f6a6e]) successfully deleted
I20260927 02:42:20.373929 2101 catalog_manager.cc:4841] TS d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993): tablet 5bc5d810747543f39f9e777231d6799b (table test-table [id=66b5dea40aeb4b1f9c1822bcee9f6a6e]) successfully deleted
Sep 27 02:42:20 dist-test-slave-44kr krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476940, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:42:20 dist-test-slave-44kr krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476940, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:20.394078 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:20.382512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.0.98.62:40341)
0927 02:42:20.382651 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:20.382652 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:20.382661 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:20.382755 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:20.382755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:20.382766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:20.382818 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:20.383359 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:20.383920 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:20.384863 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:20.385076 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:20.385141 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:20.391331 (+ 6190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:20.391334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:20.391334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:20.391348 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:20.392910 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:20.393261 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:20.393262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:20.393262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:20.393312 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:20.393578 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:20.393578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:20.393579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:20.393714 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:20.393793 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:20.393935 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 02:42:20.394003 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260927 02:42:20.395766 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:20.395949 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:20.397365 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 881308f85b464ff7b82757184eb115e2 (DEFAULT_TABLE table=test-table [id=7988bb9d8e264007a566c80c06c695e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:20.397367 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 881308f85b464ff7b82757184eb115e2 (DEFAULT_TABLE table=test-table [id=7988bb9d8e264007a566c80c06c695e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:20.397449 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 881308f85b464ff7b82757184eb115e2 (DEFAULT_TABLE table=test-table [id=7988bb9d8e264007a566c80c06c695e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:20.397500 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881308f85b464ff7b82757184eb115e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:20.397500 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881308f85b464ff7b82757184eb115e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:20.397568 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881308f85b464ff7b82757184eb115e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:20.398514 2719 tablet_bootstrap.cc:492] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: Bootstrap starting.
I20260927 02:42:20.398877 2720 tablet_bootstrap.cc:492] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: Bootstrap starting.
I20260927 02:42:20.398903 2719 tablet_bootstrap.cc:654] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:20.399199 2720 tablet_bootstrap.cc:654] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:20.399483 2721 tablet_bootstrap.cc:492] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: Bootstrap starting.
I20260927 02:42:20.399597 2719 tablet_bootstrap.cc:492] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: No bootstrap required, opened a new log
I20260927 02:42:20.399650 2719 ts_tablet_manager.cc:1397] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:20.399724 2720 tablet_bootstrap.cc:492] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: No bootstrap required, opened a new log
I20260927 02:42:20.399775 2720 ts_tablet_manager.cc:1397] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:20.399768 2719 raft_consensus.cc:348] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.399827 2721 tablet_bootstrap.cc:654] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:20.399848 2719 raft_consensus.cc:374] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:20.399871 2719 raft_consensus.cc:729] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33e8a8f637041c2bafd065c4a27e643, State: Initialized, Role: FOLLOWER
I20260927 02:42:20.399889 2720 raft_consensus.cc:348] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.399950 2720 raft_consensus.cc:374] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:20.399936 2719 consensus_queue.cc:260] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.399966 2720 raft_consensus.cc:729] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7babeff2a1b54da19f65be2c0d82e666, State: Initialized, Role: FOLLOWER
I20260927 02:42:20.400008 2720 consensus_queue.cc:260] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.400085 2720 ts_tablet_manager.cc:1428] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:20.400210 2719 ts_tablet_manager.cc:1428] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:42:20.400565 2721 tablet_bootstrap.cc:492] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: No bootstrap required, opened a new log
I20260927 02:42:20.400614 2721 ts_tablet_manager.cc:1397] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:20.400714 2721 raft_consensus.cc:348] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.400779 2721 raft_consensus.cc:374] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:20.400801 2721 raft_consensus.cc:729] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcbb82e91744384a6bc65f71bed4828, State: Initialized, Role: FOLLOWER
I20260927 02:42:20.400841 2721 consensus_queue.cc:260] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.400983 2721 ts_tablet_manager.cc:1428] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:20.519460 2725 raft_consensus.cc:482] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:20.519590 2725 raft_consensus.cc:504] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.519765 2725 leader_election.cc:288] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781), d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993)
I20260927 02:42:20.519999 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881308f85b464ff7b82757184eb115e2" candidate_uuid: "0fcbb82e91744384a6bc65f71bed4828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33e8a8f637041c2bafd065c4a27e643" is_pre_election: true
I20260927 02:42:20.520085 2413 raft_consensus.cc:2454] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fcbb82e91744384a6bc65f71bed4828 in term 0.
I20260927 02:42:20.520004 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881308f85b464ff7b82757184eb115e2" candidate_uuid: "0fcbb82e91744384a6bc65f71bed4828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7babeff2a1b54da19f65be2c0d82e666" is_pre_election: true
I20260927 02:42:20.520167 2273 raft_consensus.cc:2454] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fcbb82e91744384a6bc65f71bed4828 in term 0.
I20260927 02:42:20.520236 2485 leader_election.cc:302] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcbb82e91744384a6bc65f71bed4828, d33e8a8f637041c2bafd065c4a27e643; no voters:
I20260927 02:42:20.520361 2725 raft_consensus.cc:2781] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:20.520431 2725 raft_consensus.cc:482] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:20.520457 2725 raft_consensus.cc:3037] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:20.521193 2725 raft_consensus.cc:504] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.521368 2725 leader_election.cc:288] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [CANDIDATE]: Term 1 election: Requested vote from peers 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781), d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993)
I20260927 02:42:20.521450 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881308f85b464ff7b82757184eb115e2" candidate_uuid: "0fcbb82e91744384a6bc65f71bed4828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33e8a8f637041c2bafd065c4a27e643"
I20260927 02:42:20.521451 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881308f85b464ff7b82757184eb115e2" candidate_uuid: "0fcbb82e91744384a6bc65f71bed4828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7babeff2a1b54da19f65be2c0d82e666"
I20260927 02:42:20.521534 2273 raft_consensus.cc:3037] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:20.521533 2413 raft_consensus.cc:3037] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:20.522293 2413 raft_consensus.cc:2454] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fcbb82e91744384a6bc65f71bed4828 in term 1.
I20260927 02:42:20.522331 2273 raft_consensus.cc:2454] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fcbb82e91744384a6bc65f71bed4828 in term 1.
I20260927 02:42:20.522476 2486 leader_election.cc:302] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcbb82e91744384a6bc65f71bed4828, 7babeff2a1b54da19f65be2c0d82e666; no voters:
I20260927 02:42:20.522596 2725 raft_consensus.cc:2781] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:20.522737 2725 raft_consensus.cc:686] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 LEADER]: Becoming Leader. State: Replica: 0fcbb82e91744384a6bc65f71bed4828, State: Running, Role: LEADER
I20260927 02:42:20.522811 2725 consensus_queue.cc:237] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.523399 2116 catalog_manager.cc:5495] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0fcbb82e91744384a6bc65f71bed4828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } health_report { overall_health: HEALTHY } } }
I20260927 02:42:20.552568 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:20.550017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.0.98.3:43397)
0927 02:42:20.550234 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:20.550236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:20.550242 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:42:20.550271 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:20.550271 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:20.550281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:20.550345 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:20.550493 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:20.551037 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:20.551841 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:20.552054 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:20.552118 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:20.552266 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:20.552304 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:20.552352 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:20.552449 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 02:42:20.552475 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":146,"threads_started":1}
I20260927 02:42:20.554710 2273 raft_consensus.cc:1261] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Refusing update from remote peer 0fcbb82e91744384a6bc65f71bed4828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:20.554728 2413 raft_consensus.cc:1261] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Refusing update from remote peer 0fcbb82e91744384a6bc65f71bed4828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:20.554960 2726 consensus_queue.cc:1035] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:20.555065 2725 consensus_queue.cc:1035] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:20.565778 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:20.565896 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:20.566543 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: Sending DeleteTablet for 3 replicas of tablet 881308f85b464ff7b82757184eb115e2
I20260927 02:42:20.566792 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 881308f85b464ff7b82757184eb115e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39408
I20260927 02:42:20.566831 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 881308f85b464ff7b82757184eb115e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57462
I20260927 02:42:20.566823 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 881308f85b464ff7b82757184eb115e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37562
I20260927 02:42:20.566890 2708 tablet_replica.cc:330] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: stopping tablet replica
I20260927 02:42:20.566915 2710 tablet_replica.cc:330] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: stopping tablet replica
I20260927 02:42:20.566947 2708 raft_consensus.cc:2229] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:20.566964 2710 raft_consensus.cc:2229] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:20.567003 2710 raft_consensus.cc:2258] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:20.567008 2708 raft_consensus.cc:2258] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:20.567071 2709 tablet_replica.cc:330] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: stopping tablet replica
I20260927 02:42:20.567134 2709 raft_consensus.cc:2229] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:20.567174 2709 raft_consensus.cc:2258] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:20.567335 2710 ts_tablet_manager.cc:1905] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:20.567469 2708 ts_tablet_manager.cc:1905] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:20.568711 2709 ts_tablet_manager.cc:1905] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:20.568894 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:42:20.569007 2710 ts_tablet_manager.cc:1918] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
W20260927 02:42:20.569039 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:20.569054 2710 log.cc:1196] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/881308f85b464ff7b82757184eb115e2
I20260927 02:42:20.569036 2708 ts_tablet_manager.cc:1918] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:20.569092 2708 log.cc:1196] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/881308f85b464ff7b82757184eb115e2
I20260927 02:42:20.569243 2710 ts_tablet_manager.cc:1939] T 881308f85b464ff7b82757184eb115e2 P 7babeff2a1b54da19f65be2c0d82e666: Deleting consensus metadata
I20260927 02:42:20.569281 2708 ts_tablet_manager.cc:1939] T 881308f85b464ff7b82757184eb115e2 P 0fcbb82e91744384a6bc65f71bed4828: Deleting consensus metadata
I20260927 02:42:20.569535 2103 catalog_manager.cc:4841] TS 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397): tablet 881308f85b464ff7b82757184eb115e2 (table test-table [id=7988bb9d8e264007a566c80c06c695e0]) successfully deleted
I20260927 02:42:20.569536 2102 catalog_manager.cc:4841] TS 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781): tablet 881308f85b464ff7b82757184eb115e2 (table test-table [id=7988bb9d8e264007a566c80c06c695e0]) successfully deleted
I20260927 02:42:20.570273 2709 ts_tablet_manager.cc:1918] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:20.570333 2709 log.cc:1196] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/881308f85b464ff7b82757184eb115e2
I20260927 02:42:20.570513 2709 ts_tablet_manager.cc:1939] T 881308f85b464ff7b82757184eb115e2 P d33e8a8f637041c2bafd065c4a27e643: Deleting consensus metadata
I20260927 02:42:20.570520 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 950072ae14f04ed59400a94ea779fba9 (DEFAULT_TABLE table=test-table [id=6395691472614a0593bea5ea2c079e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:20.570555 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 950072ae14f04ed59400a94ea779fba9 (DEFAULT_TABLE table=test-table [id=6395691472614a0593bea5ea2c079e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:20.570617 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950072ae14f04ed59400a94ea779fba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:20.570642 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950072ae14f04ed59400a94ea779fba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:20.570951 2101 catalog_manager.cc:4841] TS d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993): tablet 881308f85b464ff7b82757184eb115e2 (table test-table [id=7988bb9d8e264007a566c80c06c695e0]) successfully deleted
I20260927 02:42:20.571522 2719 tablet_bootstrap.cc:492] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: Bootstrap starting.
I20260927 02:42:20.571632 2720 tablet_bootstrap.cc:492] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666: Bootstrap starting.
I20260927 02:42:20.571837 2719 tablet_bootstrap.cc:654] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:20.571928 2720 tablet_bootstrap.cc:654] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:20.572374 2719 tablet_bootstrap.cc:492] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: No bootstrap required, opened a new log
I20260927 02:42:20.572443 2719 ts_tablet_manager.cc:1397] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:20.572468 2720 tablet_bootstrap.cc:492] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666: No bootstrap required, opened a new log
I20260927 02:42:20.572505 2720 ts_tablet_manager.cc:1397] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:20.572551 2719 raft_consensus.cc:348] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.572614 2719 raft_consensus.cc:374] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:20.572631 2719 raft_consensus.cc:729] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33e8a8f637041c2bafd065c4a27e643, State: Initialized, Role: FOLLOWER
I20260927 02:42:20.572628 2720 raft_consensus.cc:348] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.572664 2719 consensus_queue.cc:260] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.572682 2720 raft_consensus.cc:374] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:20.572701 2720 raft_consensus.cc:729] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7babeff2a1b54da19f65be2c0d82e666, State: Initialized, Role: FOLLOWER
I20260927 02:42:20.572734 2720 consensus_queue.cc:260] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.572753 2719 ts_tablet_manager.cc:1428] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:20.572808 2720 ts_tablet_manager.cc:1428] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:20.573643 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 950072ae14f04ed59400a94ea779fba9 (DEFAULT_TABLE table=test-table [id=6395691472614a0593bea5ea2c079e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:20.573735 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950072ae14f04ed59400a94ea779fba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:20.574661 2721 tablet_bootstrap.cc:492] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: Bootstrap starting.
I20260927 02:42:20.574990 2721 tablet_bootstrap.cc:654] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:20.575608 2721 tablet_bootstrap.cc:492] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: No bootstrap required, opened a new log
I20260927 02:42:20.575661 2721 ts_tablet_manager.cc:1397] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:20.575752 2721 raft_consensus.cc:348] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.575819 2721 raft_consensus.cc:374] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:20.575834 2721 raft_consensus.cc:729] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcbb82e91744384a6bc65f71bed4828, State: Initialized, Role: FOLLOWER
I20260927 02:42:20.575870 2721 consensus_queue.cc:260] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.575950 2721 ts_tablet_manager.cc:1428] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:20.898732 2678 raft_consensus.cc:482] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:20.898828 2678 raft_consensus.cc:504] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.899006 2678 leader_election.cc:288] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781), 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397)
I20260927 02:42:20.899215 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950072ae14f04ed59400a94ea779fba9" candidate_uuid: "d33e8a8f637041c2bafd065c4a27e643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fcbb82e91744384a6bc65f71bed4828" is_pre_election: true
I20260927 02:42:20.899247 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950072ae14f04ed59400a94ea779fba9" candidate_uuid: "d33e8a8f637041c2bafd065c4a27e643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7babeff2a1b54da19f65be2c0d82e666" is_pre_election: true
I20260927 02:42:20.899317 2553 raft_consensus.cc:2454] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33e8a8f637041c2bafd065c4a27e643 in term 0.
I20260927 02:42:20.899324 2273 raft_consensus.cc:2454] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33e8a8f637041c2bafd065c4a27e643 in term 0.
I20260927 02:42:20.899470 2347 leader_election.cc:302] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcbb82e91744384a6bc65f71bed4828, d33e8a8f637041c2bafd065c4a27e643; no voters:
I20260927 02:42:20.899562 2678 raft_consensus.cc:2781] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:20.899598 2678 raft_consensus.cc:482] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:20.899616 2678 raft_consensus.cc:3037] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:20.938346 2678 raft_consensus.cc:504] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.938596 2678 leader_election.cc:288] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [CANDIDATE]: Term 1 election: Requested vote from peers 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781), 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397)
I20260927 02:42:20.938851 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950072ae14f04ed59400a94ea779fba9" candidate_uuid: "d33e8a8f637041c2bafd065c4a27e643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fcbb82e91744384a6bc65f71bed4828"
I20260927 02:42:20.938897 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950072ae14f04ed59400a94ea779fba9" candidate_uuid: "d33e8a8f637041c2bafd065c4a27e643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7babeff2a1b54da19f65be2c0d82e666"
I20260927 02:42:20.938962 2553 raft_consensus.cc:3037] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:20.938977 2273 raft_consensus.cc:3037] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:20.939599 2273 raft_consensus.cc:2454] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33e8a8f637041c2bafd065c4a27e643 in term 1.
I20260927 02:42:20.939735 2553 raft_consensus.cc:2454] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33e8a8f637041c2bafd065c4a27e643 in term 1.
I20260927 02:42:20.939792 2346 leader_election.cc:302] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7babeff2a1b54da19f65be2c0d82e666, d33e8a8f637041c2bafd065c4a27e643; no voters:
I20260927 02:42:20.940052 2678 raft_consensus.cc:2781] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:20.940166 2678 raft_consensus.cc:686] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 1 LEADER]: Becoming Leader. State: Replica: d33e8a8f637041c2bafd065c4a27e643, State: Running, Role: LEADER
I20260927 02:42:20.940228 2678 consensus_queue.cc:237] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } }
I20260927 02:42:20.940866 2116 catalog_manager.cc:5495] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 reported cstate change: term changed from 0 to 1, leader changed from <none> to d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d33e8a8f637041c2bafd065c4a27e643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33e8a8f637041c2bafd065c4a27e643" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:21.022809 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:21.017720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.0.98.2:34993)
0927 02:42:21.017840 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:21.017842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:21.020604 (+ 2762us) server_negotiation.cc:407] Connection header received
0927 02:42:21.020715 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:21.020716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:21.020729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:21.020803 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:21.020914 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:21.021422 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:21.022162 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:21.022368 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:21.022422 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:21.022556 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:21.022586 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:21.022633 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:21.022690 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 02:42:21.022717 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260927 02:42:21.025437 2273 raft_consensus.cc:1261] T 950072ae14f04ed59400a94ea779fba9 P 7babeff2a1b54da19f65be2c0d82e666 [term 1 FOLLOWER]: Refusing update from remote peer d33e8a8f637041c2bafd065c4a27e643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:21.025419 2553 raft_consensus.cc:1261] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Refusing update from remote peer d33e8a8f637041c2bafd065c4a27e643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:21.025704 2678 consensus_queue.cc:1035] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fcbb82e91744384a6bc65f71bed4828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:21.025843 2737 consensus_queue.cc:1035] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7babeff2a1b54da19f65be2c0d82e666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:42:21.030480 2690 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:42:22.548611 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:22.548717 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:22.549541 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dd11ad2ca1840a787face04fa33ebe3: Sending DeleteTablet for 3 replicas of tablet 950072ae14f04ed59400a94ea779fba9
I20260927 02:42:22.549841 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 950072ae14f04ed59400a94ea779fba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57462
I20260927 02:42:22.549846 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 950072ae14f04ed59400a94ea779fba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39408
I20260927 02:42:22.549839 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 950072ae14f04ed59400a94ea779fba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37562
I20260927 02:42:22.549942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 2188
I20260927 02:42:22.550676 2767 tablet_replica.cc:330] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: stopping tablet replica
I20260927 02:42:22.550736 2767 raft_consensus.cc:2229] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:22.550776 2767 raft_consensus.cc:2258] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:22.551007 2767 ts_tablet_manager.cc:1905] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:22.551896 2769 tablet_replica.cc:330] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: stopping tablet replica
I20260927 02:42:22.551968 2769 raft_consensus.cc:2229] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:22.552028 2769 raft_consensus.cc:2258] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:22.552251 2769 ts_tablet_manager.cc:1905] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:22.552649 2767 ts_tablet_manager.cc:1918] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:42:22.552700 2767 log.cc:1196] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/950072ae14f04ed59400a94ea779fba9
I20260927 02:42:22.552881 2767 ts_tablet_manager.cc:1939] T 950072ae14f04ed59400a94ea779fba9 P 0fcbb82e91744384a6bc65f71bed4828: Deleting consensus metadata
I20260927 02:42:22.553208 2103 catalog_manager.cc:4841] TS 0fcbb82e91744384a6bc65f71bed4828 (127.0.98.3:43397): tablet 950072ae14f04ed59400a94ea779fba9 (table test-table [id=6395691472614a0593bea5ea2c079e98]) successfully deleted
I20260927 02:42:22.554123 2769 ts_tablet_manager.cc:1918] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:42:22.554183 2769 log.cc:1196] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/950072ae14f04ed59400a94ea779fba9
I20260927 02:42:22.554401 2769 ts_tablet_manager.cc:1939] T 950072ae14f04ed59400a94ea779fba9 P d33e8a8f637041c2bafd065c4a27e643: Deleting consensus metadata
I20260927 02:42:22.554701 2101 catalog_manager.cc:4841] TS d33e8a8f637041c2bafd065c4a27e643 (127.0.98.2:34993): tablet 950072ae14f04ed59400a94ea779fba9 (table test-table [id=6395691472614a0593bea5ea2c079e98]) successfully deleted
W20260927 02:42:22.556238 2102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41781 (error 108)
I20260927 02:42:22.556334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 2328
I20260927 02:42:22.556965 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:22.556653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41781 (local address 127.0.0.1:55062)
0927 02:42:22.556880 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:22.556927 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
W20260927 02:42:22.557101 2102 catalog_manager.cc:4584] TS 7babeff2a1b54da19f65be2c0d82e666 (127.0.98.1:41781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 950072ae14f04ed59400a94ea779fba9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41781: connect: Connection refused (error 111)
I20260927 02:42:22.562126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 2468
I20260927 02:42:22.567477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 2084
2026-09-27T02:42:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6200 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:22 dist-test-slave-44kr krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:22 dist-test-slave-44kr krb5kdc[2774](info): set up 2 sockets
Sep 27 02:42:22 dist-test-slave-44kr krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260927 02:42:23.532006 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:23 dist-test-slave-44kr krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:23.649542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:23.729525 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:23.729763 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:23.729796 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:23.730964 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:23.731001 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:23.731014 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:23.731025 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:23.731037 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:23.732580 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:23.732800 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:23.732980 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:23.738571 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:23.738590 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:23.738590 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:23.738646 2787 server_base.cc:1034] running on GCE node
I20260927 02:42:23.738988 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:23.739320 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:23.740464 2787 hybrid_clock.cc:648] HybridClock initialized: now 1790476943740440 us; error 37 us; skew 500 ppm
Sep 27 02:42:23 dist-test-slave-44kr krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:23.743116 2787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:23.743685 2787 webserver.cc:460] Webserver started at http://127.0.98.62:33013/ using document root <none> and password file <none>
I20260927 02:42:23.743911 2787 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:23.743956 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:23.744083 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:23.745268 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "e4d755c711a3413db757fc4318cea16a"
format_stamp: "Formatted at 2026-09-27 02:42:23 on dist-test-slave-44kr"
server_key: "5983937114d79dab8151f5ff5070c831"
server_key_iv: "372541f08bcf9a2a8e1df9d7216c55c7"
server_key_version: "encryptionkey@0"
I20260927 02:42:23.745615 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4d755c711a3413db757fc4318cea16a"
format_stamp: "Formatted at 2026-09-27 02:42:23 on dist-test-slave-44kr"
server_key: "5983937114d79dab8151f5ff5070c831"
server_key_iv: "372541f08bcf9a2a8e1df9d7216c55c7"
server_key_version: "encryptionkey@0"
I20260927 02:42:23.746996 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:42:23.747897 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:23.748195 2787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:23.748368 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "e4d755c711a3413db757fc4318cea16a"
format_stamp: "Formatted at 2026-09-27 02:42:23 on dist-test-slave-44kr"
server_key: "5983937114d79dab8151f5ff5070c831"
server_key_iv: "372541f08bcf9a2a8e1df9d7216c55c7"
server_key_version: "encryptionkey@0"
I20260927 02:42:23.748478 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:23.758947 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:23.761152 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:23.761293 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:23.765324 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38287
I20260927 02:42:23.765362 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38287 every 8 connection(s)
I20260927 02:42:23.765836 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:23.766510 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:23.768788 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a: Bootstrap starting.
I20260927 02:42:23.769343 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:23.769603 2855 log.cc:824] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:23.770292 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a: No bootstrap required, opened a new log
I20260927 02:42:23.771906 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d755c711a3413db757fc4318cea16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38287 } }
I20260927 02:42:23.772059 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:23.772092 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d755c711a3413db757fc4318cea16a, State: Initialized, Role: FOLLOWER
I20260927 02:42:23.772254 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d755c711a3413db757fc4318cea16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38287 } }
I20260927 02:42:23.772320 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:23.772348 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:23.772418 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:23.773288 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d755c711a3413db757fc4318cea16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38287 } }
I20260927 02:42:23.773379 2855 leader_election.cc:302] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4d755c711a3413db757fc4318cea16a; no voters:
I20260927 02:42:23.773536 2855 leader_election.cc:288] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:23.773593 2858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:23.773747 2858 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [term 1 LEADER]: Becoming Leader. State: Replica: e4d755c711a3413db757fc4318cea16a, State: Running, Role: LEADER
I20260927 02:42:23.773787 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:23.773907 2858 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d755c711a3413db757fc4318cea16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38287 } }
I20260927 02:42:23.774057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 2787
I20260927 02:42:23.774142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:23.774253 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4d755c711a3413db757fc4318cea16a. Latest consensus state: current_term: 1 leader_uuid: "e4d755c711a3413db757fc4318cea16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d755c711a3413db757fc4318cea16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38287 } } }
I20260927 02:42:23.774318 392 external_mini_cluster.cc:1428] Setting key 73a9b95b3efdb781ab7bdfd57a5ae21b
I20260927 02:42:23.774456 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:23.774399 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4d755c711a3413db757fc4318cea16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d755c711a3413db757fc4318cea16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38287 } } }
I20260927 02:42:23.774595 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:23.774698 2867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:23.775532 2867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:23.777105 2867 catalog_manager.cc:1347] Generated new cluster ID: 0fd18bcdfd4e45c0983c873debeb52d4
I20260927 02:42:23.777153 2867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:42:23 dist-test-slave-44kr krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:23.782302 2867 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:23.782963 2867 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:23.784246 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:23.774981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.0.98.62:38287)
0927 02:42:23.775352 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:23.775361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:23.775396 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:42:23.775491 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:23.775497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:23.775541 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:23.775709 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:23.776308 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:23.777295 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:23.778023 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:23.778501 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:23.778595 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:23.780728 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:23.780732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:23.780732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:23.780745 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:23.782843 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:23.783173 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:23.783176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:23.783176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:23.783269 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:23.783446 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:23.783447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:23.783448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:23.783663 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:23.783829 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:23.783897 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:42:23.784046 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":53,"threads_started":1}
I20260927 02:42:23.789590 2867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4d755c711a3413db757fc4318cea16a: Generated new TSK 0
I20260927 02:42:23.789794 2867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:23.825023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38287
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:23.914579 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:23.914834 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:23.914860 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:23.916142 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:23.916203 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:23.916236 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:23.917876 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:23.918112 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:23.918345 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:23.923300 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:23.923334 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:23.923329 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:23.923606 2881 server_base.cc:1034] running on GCE node
I20260927 02:42:23.923812 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:23.924063 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:23.925235 2881 hybrid_clock.cc:648] HybridClock initialized: now 1790476943925203 us; error 27 us; skew 500 ppm
Sep 27 02:42:23 dist-test-slave-44kr krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:23.927668 2881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:23.928122 2881 webserver.cc:460] Webserver started at http://127.0.98.1:34083/ using document root <none> and password file <none>
I20260927 02:42:23.928331 2881 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:23.928380 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:23.928537 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:23.929576 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "773d97c81b6f4dd68e7e4aefe0365048"
format_stamp: "Formatted at 2026-09-27 02:42:23 on dist-test-slave-44kr"
server_key: "cbb8b1ef4284bf8c1d094a276b3b558e"
server_key_iv: "752354f4f28f3d4c721cf0ef0247f397"
server_key_version: "encryptionkey@0"
I20260927 02:42:23.929903 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "773d97c81b6f4dd68e7e4aefe0365048"
format_stamp: "Formatted at 2026-09-27 02:42:23 on dist-test-slave-44kr"
server_key: "cbb8b1ef4284bf8c1d094a276b3b558e"
server_key_iv: "752354f4f28f3d4c721cf0ef0247f397"
server_key_version: "encryptionkey@0"
I20260927 02:42:23.931125 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:42:23.931864 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:23.932015 2881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:23.932093 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "773d97c81b6f4dd68e7e4aefe0365048"
format_stamp: "Formatted at 2026-09-27 02:42:23 on dist-test-slave-44kr"
server_key: "cbb8b1ef4284bf8c1d094a276b3b558e"
server_key_iv: "752354f4f28f3d4c721cf0ef0247f397"
server_key_version: "encryptionkey@0"
I20260927 02:42:23.932157 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:23.940332 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:23.942425 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:23.942559 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:23.942770 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:23.943054 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:23.943085 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:23.943113 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:23.943135 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:23.948868 2881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43219
I20260927 02:42:23.948915 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43219 every 8 connection(s)
I20260927 02:42:23.949254 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:23.950352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 2881
I20260927 02:42:23.950434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:23.950588 392 external_mini_cluster.cc:1428] Setting key e1929bc568ae95a63723600d41117fa4
Sep 27 02:42:23 dist-test-slave-44kr krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:42:23.959296 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:23.949948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43367 (local address 127.0.98.62:38287)
0927 02:42:23.950027 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:23.950029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:23.950186 (+ 157us) server_negotiation.cc:407] Connection header received
0927 02:42:23.951271 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:23.951272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:23.951284 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:23.951330 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:23.952008 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:23.952631 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:23.953733 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:23.953931 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:23.953985 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:23.956339 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:23.956341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:23.956341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:23.956349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:23.957802 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:23.958221 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:23.958223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:23.958223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:23.958282 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:23.958630 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:23.958631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:23.958632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:23.958798 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:23.958871 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:23.959149 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 02:42:23.959229 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:42:23.959508 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:23.949921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38287 (local address 127.0.98.1:43367)
0927 02:42:23.950072 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:23.950109 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:23.951170 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:23.951407 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:23.951412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:23.951710 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:23.951927 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:23.951931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:23.952724 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:23.952725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:23.953647 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:23.953649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:23.954005 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:23.954006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:23.954140 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:23.954681 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:23.954689 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:23.956214 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:23.957902 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:23.957908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:23.957910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:23.958138 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:23.958407 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:23.958408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:23.958408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:23.958547 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:23.958878 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:23.958884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:23.959024 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 02:42:23.959115 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:42:23.959302 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260927 02:42:23.960011 3010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38287
I20260927 02:42:23.960108 3010 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:23.960268 3010 heartbeater.cc:511] Master 127.0.98.62:38287 requested a full tablet report, sending...
I20260927 02:42:23.960807 2819 ts_manager.cc:194] Registered new tserver with Master: 773d97c81b6f4dd68e7e4aefe0365048 (127.0.98.1:43219)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:42:23.961452 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:23.978823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38287
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:24.060659 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:24.060866 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:24.060889 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:24.062103 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:24.062156 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:24.062182 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:24.063766 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:24.063973 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:24.064157 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:24.069545 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:24.069543 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:24.069563 3015 server_base.cc:1034] running on GCE node
W20260927 02:42:24.069545 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:24.070016 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:24.070257 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:24.071413 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790476944071382 us; error 48 us; skew 500 ppm
Sep 27 02:42:24 dist-test-slave-44kr krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:24.074102 3015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:24.074656 3015 webserver.cc:460] Webserver started at http://127.0.98.2:46099/ using document root <none> and password file <none>
I20260927 02:42:24.074918 3015 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:24.074978 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:24.075107 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:24.076210 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "025597424e8747fcbdd366f7cab024ae"
format_stamp: "Formatted at 2026-09-27 02:42:24 on dist-test-slave-44kr"
server_key: "7b152bd1cca11095747ef031264d4fc4"
server_key_iv: "311a936f52e3098d146688a4a426e608"
server_key_version: "encryptionkey@0"
I20260927 02:42:24.076581 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "025597424e8747fcbdd366f7cab024ae"
format_stamp: "Formatted at 2026-09-27 02:42:24 on dist-test-slave-44kr"
server_key: "7b152bd1cca11095747ef031264d4fc4"
server_key_iv: "311a936f52e3098d146688a4a426e608"
server_key_version: "encryptionkey@0"
I20260927 02:42:24.077917 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:24.078783 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:24.078953 3015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:24.079041 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "025597424e8747fcbdd366f7cab024ae"
format_stamp: "Formatted at 2026-09-27 02:42:24 on dist-test-slave-44kr"
server_key: "7b152bd1cca11095747ef031264d4fc4"
server_key_iv: "311a936f52e3098d146688a4a426e608"
server_key_version: "encryptionkey@0"
I20260927 02:42:24.079113 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:24.104949 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:24.107362 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:24.107518 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:24.107766 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:24.108073 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:24.108108 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:24.108137 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:24.108155 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:24.113544 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46263
I20260927 02:42:24.113595 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46263 every 8 connection(s)
I20260927 02:42:24.113898 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:24.114032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 3015
I20260927 02:42:24.114120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:24.114322 392 external_mini_cluster.cc:1428] Setting key 513f01fbe68b3abf5e54da1b0c6765ee
Sep 27 02:42:24 dist-test-slave-44kr krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:42:24.124832 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:24.114689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41823 (local address 127.0.98.62:38287)
0927 02:42:24.114733 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:24.114735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:24.114973 (+ 238us) server_negotiation.cc:407] Connection header received
0927 02:42:24.115937 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:24.115938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:24.115951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:24.116023 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:24.116891 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:24.117442 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:24.118919 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:24.119162 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:24.119223 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:24.121910 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:24.121911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:24.121912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:24.121919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:24.123429 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:24.123812 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:24.123813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:24.123813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:24.123869 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:24.124142 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:24.124143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:24.124144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:24.124280 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:24.124335 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:24.124637 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 02:42:24.124748 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:42:24.124917 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:24.114668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38287 (local address 127.0.98.2:41823)
0927 02:42:24.114836 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:24.114884 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:24.115809 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:24.116080 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:24.116084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:24.116488 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:24.116778 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:24.116784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:24.117540 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:24.117541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:24.118810 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:24.118812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:24.119253 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:24.119254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:24.119410 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:24.120030 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:24.120040 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:24.121809 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:24.123501 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:24.123507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:24.123509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:24.123723 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:24.123926 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:24.123926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:24.123926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:24.124016 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:24.124333 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:24.124338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:24.124497 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 02:42:24.124586 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:42:24.124714 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260927 02:42:24.125502 3144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38287
I20260927 02:42:24.125599 3144 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:24.125777 3144 heartbeater.cc:511] Master 127.0.98.62:38287 requested a full tablet report, sending...
I20260927 02:42:24.126205 2819 ts_manager.cc:194] Registered new tserver with Master: 025597424e8747fcbdd366f7cab024ae (127.0.98.2:46263)
I20260927 02:42:24.126719 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:24.143606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38287
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:24.222855 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:24.223049 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:24.223070 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:24.224195 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:24.224239 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:24.224264 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:24.225796 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:24.225996 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:24.226197 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:24.231547 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:24.231626 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:24.231550 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:24.231854 3149 server_base.cc:1034] running on GCE node
I20260927 02:42:24.232036 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:24.232286 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:24.233448 3149 hybrid_clock.cc:648] HybridClock initialized: now 1790476944233422 us; error 38 us; skew 500 ppm
Sep 27 02:42:24 dist-test-slave-44kr krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:24.236537 3149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:24.236991 3149 webserver.cc:460] Webserver started at http://127.0.98.3:40835/ using document root <none> and password file <none>
I20260927 02:42:24.237221 3149 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:24.237270 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:24.237402 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:24.238492 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "8f7ae7fb6e07432aaabff08ab8864383"
format_stamp: "Formatted at 2026-09-27 02:42:24 on dist-test-slave-44kr"
server_key: "84cd34e38aa172360b96f24e60bf48b6"
server_key_iv: "bcd9e16f24c9838898c0f96dafc96fc1"
server_key_version: "encryptionkey@0"
I20260927 02:42:24.238826 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8f7ae7fb6e07432aaabff08ab8864383"
format_stamp: "Formatted at 2026-09-27 02:42:24 on dist-test-slave-44kr"
server_key: "84cd34e38aa172360b96f24e60bf48b6"
server_key_iv: "bcd9e16f24c9838898c0f96dafc96fc1"
server_key_version: "encryptionkey@0"
I20260927 02:42:24.240118 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:24.240970 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:24.241272 3149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:42:24.241374 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "8f7ae7fb6e07432aaabff08ab8864383"
format_stamp: "Formatted at 2026-09-27 02:42:24 on dist-test-slave-44kr"
server_key: "84cd34e38aa172360b96f24e60bf48b6"
server_key_iv: "bcd9e16f24c9838898c0f96dafc96fc1"
server_key_version: "encryptionkey@0"
I20260927 02:42:24.241446 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:24.261620 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:24.263594 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:24.263748 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:24.263984 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:24.264304 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:24.264338 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:24.264369 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:24.264410 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:24.270030 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40401
I20260927 02:42:24.270087 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40401 every 8 connection(s)
I20260927 02:42:24.270398 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:42:24 dist-test-slave-44kr krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:24.278962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 3149
I20260927 02:42:24.279067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:24.279268 392 external_mini_cluster.cc:1428] Setting key aee71ec9a08b581c21bcd8644a95629c
I20260927 02:42:24.280277 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:24.271146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36985 (local address 127.0.98.62:38287)
0927 02:42:24.271198 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:24.271200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:24.271365 (+ 165us) server_negotiation.cc:407] Connection header received
0927 02:42:24.272277 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:24.272278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:24.272290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:24.272366 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:24.273113 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:24.273613 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:24.274857 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:24.275073 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:24.275182 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:24.277431 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:24.277432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:24.277433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:24.277440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:24.278702 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:24.279228 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:24.279229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:24.279230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:24.279289 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:24.279570 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:24.279571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:24.279571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:24.279709 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:24.279770 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:24.280095 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 02:42:24.280182 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:42:24.280479 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:24.271087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38287 (local address 127.0.98.3:36985)
0927 02:42:24.271238 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:24.271280 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:24.272155 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:24.272500 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:24.272512 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:24.272796 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:24.272995 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:24.272999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:24.273718 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:24.273718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:24.274739 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:24.274741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:24.275222 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:24.275223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:24.275363 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:24.275889 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:24.275897 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:24.277342 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:24.278793 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:24.278800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:24.278803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:24.279098 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:24.279363 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:24.279364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:24.279364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:24.279502 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:24.279778 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:24.279784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:24.279956 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 02:42:24.280055 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:42:24.280219 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260927 02:42:24.280918 3278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38287
I20260927 02:42:24.281033 3278 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:24.281215 3278 heartbeater.cc:511] Master 127.0.98.62:38287 requested a full tablet report, sending...
I20260927 02:42:24.281563 2819 ts_manager.cc:194] Registered new tserver with Master: 8f7ae7fb6e07432aaabff08ab8864383 (127.0.98.3:40401)
I20260927 02:42:24.281966 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36985
I20260927 02:42:24.291278 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:42:24.301990 2875 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:24.301058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.0.98.62:38287)
0927 02:42:24.301140 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:24.301142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:24.301209 (+ 67us) server_negotiation.cc:407] Connection header received
0927 02:42:24.301289 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:24.301290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:24.301300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:24.301341 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:24.301924 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44692: BlockingRecv error: recv got EOF from 127.0.0.1:44692 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:42:24.302836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 2881
I20260927 02:42:24.307225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 3015
I20260927 02:42:24.311391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 3149
I20260927 02:42:24.315181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 2787
2026-09-27T02:42:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1746 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:24 dist-test-slave-44kr krb5kdc[3290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:24 dist-test-slave-44kr krb5kdc[3290](info): set up 2 sockets
Sep 27 02:42:24 dist-test-slave-44kr krb5kdc[3290](info): commencing operation
krb5kdc: starting...
W20260927 02:42:25.280210 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:25 dist-test-slave-44kr krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:25.400017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33333
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:25.479173 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:25.479383 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:25.479404 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:25.480621 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:25.480672 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:25.480685 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:25.480696 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:25.480707 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:25.482266 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:25.482499 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:25.482802 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:25.488020 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:25.488039 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:25.488181 3303 server_base.cc:1034] running on GCE node
W20260927 02:42:25.488020 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:25.488524 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:25.488834 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:25.490000 3303 hybrid_clock.cc:648] HybridClock initialized: now 1790476945489983 us; error 32 us; skew 500 ppm
Sep 27 02:42:25 dist-test-slave-44kr krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:25.503389 3303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:25.503924 3303 webserver.cc:460] Webserver started at http://127.0.98.62:36781/ using document root <none> and password file <none>
I20260927 02:42:25.504184 3303 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:25.504251 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:25.504405 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:25.505765 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "e9b72fc051334f50ac09aab6f5f4c3a3"
format_stamp: "Formatted at 2026-09-27 02:42:25 on dist-test-slave-44kr"
server_key: "034144b9b312eedb2fe37d4a60a6b178"
server_key_iv: "e794422e4dfaa0533dfd1dc9e7c27498"
server_key_version: "encryptionkey@0"
I20260927 02:42:25.506145 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9b72fc051334f50ac09aab6f5f4c3a3"
format_stamp: "Formatted at 2026-09-27 02:42:25 on dist-test-slave-44kr"
server_key: "034144b9b312eedb2fe37d4a60a6b178"
server_key_iv: "e794422e4dfaa0533dfd1dc9e7c27498"
server_key_version: "encryptionkey@0"
I20260927 02:42:25.507578 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:25.508512 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:25.508736 3303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:25.508831 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "e9b72fc051334f50ac09aab6f5f4c3a3"
format_stamp: "Formatted at 2026-09-27 02:42:25 on dist-test-slave-44kr"
server_key: "034144b9b312eedb2fe37d4a60a6b178"
server_key_iv: "e794422e4dfaa0533dfd1dc9e7c27498"
server_key_version: "encryptionkey@0"
I20260927 02:42:25.508913 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:25.547302 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:25.549458 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:25.549606 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:25.553303 3303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33333
I20260927 02:42:25.553351 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33333 every 8 connection(s)
I20260927 02:42:25.553668 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:25.554531 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:25.555032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 3303
I20260927 02:42:25.555115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:25.555289 392 external_mini_cluster.cc:1428] Setting key 296b6e939938c4f105c957604a8c9b52
I20260927 02:42:25.557525 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3: Bootstrap starting.
I20260927 02:42:25.558219 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:25.558499 3371 log.cc:824] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:25.559309 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3: No bootstrap required, opened a new log
I20260927 02:42:25.561419 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b72fc051334f50ac09aab6f5f4c3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33333 } }
I20260927 02:42:25.561551 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:25.561573 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b72fc051334f50ac09aab6f5f4c3a3, State: Initialized, Role: FOLLOWER
I20260927 02:42:25.561681 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b72fc051334f50ac09aab6f5f4c3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33333 } }
I20260927 02:42:25.561751 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:25.561776 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:25.561821 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:42:25 dist-test-slave-44kr krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:25.562450 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b72fc051334f50ac09aab6f5f4c3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33333 } }
I20260927 02:42:25.562539 3371 leader_election.cc:302] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b72fc051334f50ac09aab6f5f4c3a3; no voters:
I20260927 02:42:25.562711 3371 leader_election.cc:288] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:25.562754 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:25.562901 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [term 1 LEADER]: Becoming Leader. State: Replica: e9b72fc051334f50ac09aab6f5f4c3a3, State: Running, Role: LEADER
I20260927 02:42:25.562970 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:25.562995 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b72fc051334f50ac09aab6f5f4c3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33333 } }
I20260927 02:42:25.563359 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b72fc051334f50ac09aab6f5f4c3a3. Latest consensus state: current_term: 1 leader_uuid: "e9b72fc051334f50ac09aab6f5f4c3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b72fc051334f50ac09aab6f5f4c3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33333 } } }
I20260927 02:42:25.563443 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:25.563657 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b72fc051334f50ac09aab6f5f4c3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b72fc051334f50ac09aab6f5f4c3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33333 } } }
I20260927 02:42:25.563786 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:25.563973 3383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:25.564411 3383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:25.567072 3383 catalog_manager.cc:1347] Generated new cluster ID: 13e16b2cd80f4dc6b4d84ef8d0618d08
I20260927 02:42:25.567119 3383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:25.567894 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:25.555767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.0.98.62:33333)
0927 02:42:25.555950 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:25.555960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:25.555995 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:42:25.556559 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:25.556568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:25.556632 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:25.556782 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:25.557368 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:25.558713 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:25.559613 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:25.560239 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:25.560353 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:25.562678 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:25.562681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:25.562682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:25.562692 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:25.566048 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:25.566483 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:25.566487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:25.566487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:25.566590 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:25.566830 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:25.566831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:25.566831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:25.567143 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:25.567294 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:25.567551 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:42:25.567687 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260927 02:42:25.568792 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:25.589385 3383 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:25.589811 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:25.585223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.0.98.62:33333)
0927 02:42:25.585270 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:25.585272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:25.585431 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:42:25.585504 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:25.585505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:25.585519 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:25.585574 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:25.586026 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:25.586773 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:25.587942 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:25.588282 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:25.588350 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:25.589227 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:25.589229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:25.589229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:25.589469 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:25.589491 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:25.589499 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:25.589555 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:25.589691 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:42:25.589725 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":25}
I20260927 02:42:25.590117 3383 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:25.598991 3383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b72fc051334f50ac09aab6f5f4c3a3: Generated new TSK 0
I20260927 02:42:25.599248 3383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:25.602952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 3303
2026-09-27T02:42:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1285 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:25 dist-test-slave-44kr krb5kdc[3404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:25 dist-test-slave-44kr krb5kdc[3404](info): set up 2 sockets
Sep 27 02:42:25 dist-test-slave-44kr krb5kdc[3404](info): commencing operation
krb5kdc: starting...
W20260927 02:42:26.574558 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:26 dist-test-slave-44kr krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:26.699369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39033
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:26.779352 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:26.779579 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:26.779601 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:26.780819 3417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:26.780864 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:26.780876 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:26.780889 3417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:26.780898 3417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:26.782400 3417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:26.782635 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:26.782850 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:26.787909 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:26.787909 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:26.787911 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:26.788364 3417 server_base.cc:1034] running on GCE node
I20260927 02:42:26.788633 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:26.788950 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:26.790108 3417 hybrid_clock.cc:648] HybridClock initialized: now 1790476946790085 us; error 51 us; skew 500 ppm
Sep 27 02:42:26 dist-test-slave-44kr krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:26.792898 3417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:26.793383 3417 webserver.cc:460] Webserver started at http://127.0.98.62:35381/ using document root <none> and password file <none>
I20260927 02:42:26.793716 3417 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:26.793776 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:26.793905 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:26.794962 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8"
format_stamp: "Formatted at 2026-09-27 02:42:26 on dist-test-slave-44kr"
server_key: "461d97834d27123a284d2aaa399205b9"
server_key_iv: "cd89fa1e33a8ca8f39596f3db1e73836"
server_key_version: "encryptionkey@0"
I20260927 02:42:26.795295 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8"
format_stamp: "Formatted at 2026-09-27 02:42:26 on dist-test-slave-44kr"
server_key: "461d97834d27123a284d2aaa399205b9"
server_key_iv: "cd89fa1e33a8ca8f39596f3db1e73836"
server_key_version: "encryptionkey@0"
I20260927 02:42:26.796641 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:42:26.797370 3432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:26.797519 3417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:26.797616 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8"
format_stamp: "Formatted at 2026-09-27 02:42:26 on dist-test-slave-44kr"
server_key: "461d97834d27123a284d2aaa399205b9"
server_key_iv: "cd89fa1e33a8ca8f39596f3db1e73836"
server_key_version: "encryptionkey@0"
I20260927 02:42:26.797684 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:26.809947 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:26.812237 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:26.812409 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:26.816625 3417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39033
I20260927 02:42:26.816648 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39033 every 8 connection(s)
I20260927 02:42:26.817037 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:26.818006 3485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:26.820515 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8: Bootstrap starting.
I20260927 02:42:26.821108 3485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:26.821373 3485 log.cc:824] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:26.822240 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8: No bootstrap required, opened a new log
I20260927 02:42:26.823920 3485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39033 } }
I20260927 02:42:26.824079 3485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:26.824111 3485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97aa6d5e863f420d97e4ddbc8f0de7b8, State: Initialized, Role: FOLLOWER
I20260927 02:42:26.824258 3485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39033 } }
I20260927 02:42:26.824309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 3417
I20260927 02:42:26.824337 3485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:26.824371 3485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:26.824419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:26.824443 3485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:26.824587 392 external_mini_cluster.cc:1428] Setting key 6c37bda9670d38100267008013b82f93
I20260927 02:42:26.825563 3485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39033 } }
I20260927 02:42:26.825677 3485 leader_election.cc:302] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97aa6d5e863f420d97e4ddbc8f0de7b8; no voters:
I20260927 02:42:26.825875 3485 leader_election.cc:288] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:26.825897 3490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:26.826001 3490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [term 1 LEADER]: Becoming Leader. State: Replica: 97aa6d5e863f420d97e4ddbc8f0de7b8, State: Running, Role: LEADER
I20260927 02:42:26.826138 3490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39033 } }
I20260927 02:42:26.826153 3485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:26.826509 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39033 } } }
I20260927 02:42:26.826582 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:26.826741 3492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97aa6d5e863f420d97e4ddbc8f0de7b8. Latest consensus state: current_term: 1 leader_uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97aa6d5e863f420d97e4ddbc8f0de7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39033 } } }
I20260927 02:42:26.826828 3492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8 [sys.catalog]: This master's current role is: LEADER
W20260927 02:42:26.828078 3506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:26.828132 3506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:26.828183 3494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:26.828338 3494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:26.830005 3494 catalog_manager.cc:1347] Generated new cluster ID: 5ef3e0b2d32d49aab5c34c71cd262358
I20260927 02:42:26.830055 3494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:42:26 dist-test-slave-44kr krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:26.834231 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:26.825063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.0.98.62:39033)
0927 02:42:26.825334 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:26.825345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:26.825383 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:42:26.825508 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:26.825515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:26.825579 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:26.825744 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:26.826428 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:26.827659 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:26.828554 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:26.828981 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:26.829074 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:26.831321 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:26.831326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:26.831326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:26.831338 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:26.832870 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:26.833184 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:26.833187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:26.833188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:26.833281 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:26.833453 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:26.833454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:26.833454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:26.833684 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:26.833827 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:26.833895 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:42:26.834023 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":167,"threads_started":1}
I20260927 02:42:26.843315 3494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:26.843840 3494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:26.846040 3494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97aa6d5e863f420d97e4ddbc8f0de7b8: Generated new TSK 0
I20260927 02:42:26.846197 3494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:26.875829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39033
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:26.957603 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:26.957832 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:26.957856 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:26.959060 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:26.959118 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:26.959146 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:26.960718 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:26.960969 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:26.961220 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:26.966125 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:26.966143 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:26.966364 3511 server_base.cc:1034] running on GCE node
W20260927 02:42:26.966136 3517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:26.966655 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:26.966922 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:26.968075 3511 hybrid_clock.cc:648] HybridClock initialized: now 1790476946968061 us; error 35 us; skew 500 ppm
Sep 27 02:42:26 dist-test-slave-44kr krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:26.970712 3511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:26.971168 3511 webserver.cc:460] Webserver started at http://127.0.98.1:44413/ using document root <none> and password file <none>
I20260927 02:42:26.971375 3511 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:26.971417 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:26.971513 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:26.972570 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "6b852b5ee09e4b55920fa6d689249873"
format_stamp: "Formatted at 2026-09-27 02:42:26 on dist-test-slave-44kr"
server_key: "264fdffb2f7a30a92bce9222c794fc19"
server_key_iv: "425c23d70702e737bc4b8d244ddbf632"
server_key_version: "encryptionkey@0"
I20260927 02:42:26.972883 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6b852b5ee09e4b55920fa6d689249873"
format_stamp: "Formatted at 2026-09-27 02:42:26 on dist-test-slave-44kr"
server_key: "264fdffb2f7a30a92bce9222c794fc19"
server_key_iv: "425c23d70702e737bc4b8d244ddbf632"
server_key_version: "encryptionkey@0"
I20260927 02:42:26.974151 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:26.974891 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:26.975142 3511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:26.975224 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "6b852b5ee09e4b55920fa6d689249873"
format_stamp: "Formatted at 2026-09-27 02:42:26 on dist-test-slave-44kr"
server_key: "264fdffb2f7a30a92bce9222c794fc19"
server_key_iv: "425c23d70702e737bc4b8d244ddbf632"
server_key_version: "encryptionkey@0"
I20260927 02:42:26.975301 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:26.984156 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:26.986207 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:26.986335 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:26.986591 3511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:26.986901 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:26.986934 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:26.986958 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:26.986982 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:26.992516 3511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46129
I20260927 02:42:26.992597 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46129 every 8 connection(s)
I20260927 02:42:26.992889 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:42:26 dist-test-slave-44kr krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:27.000494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 3511
I20260927 02:42:27.000588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:27.000818 392 external_mini_cluster.cc:1428] Setting key 0c65f5d105501a8301e4b808edbed633
I20260927 02:42:27.002485 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:26.993668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50773 (local address 127.0.98.62:39033)
0927 02:42:26.993738 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:26.993740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:26.993916 (+ 176us) server_negotiation.cc:407] Connection header received
0927 02:42:26.994422 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:26.994423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:26.994435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:26.994492 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:26.995458 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:26.995967 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:26.997130 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:26.997328 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:26.997385 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:26.999626 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:26.999627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:26.999627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:26.999634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:27.001122 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:27.001600 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:27.001601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:27.001602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:27.001668 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:27.001880 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:27.001881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:27.001882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:27.001992 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:27.002046 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:27.002315 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 02:42:27.002398 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:42:27.002665 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:26.993632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39033 (local address 127.0.98.1:50773)
0927 02:42:26.993788 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:26.993828 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:26.994309 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:26.994558 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:26.994561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:26.995142 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:26.995332 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:26.995336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:26.996117 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:26.996118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:26.997023 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:26.997026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:26.997425 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:26.997426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:26.997554 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:26.998169 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:26.998178 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:26.999521 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:27.001229 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:27.001236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:27.001238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:27.001536 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:27.001719 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:27.001720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:27.001720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:27.001828 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:27.002045 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:27.002052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:27.002196 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 02:42:27.002287 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:42:27.002441 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260927 02:42:27.003201 3640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39033
I20260927 02:42:27.003290 3640 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:27.003454 3640 heartbeater.cc:511] Master 127.0.98.62:39033 requested a full tablet report, sending...
I20260927 02:42:27.003947 3449 ts_manager.cc:194] Registered new tserver with Master: 6b852b5ee09e4b55920fa6d689249873 (127.0.98.1:46129)
I20260927 02:42:27.004591 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50773
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:27.027844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39033
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:27.109688 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:27.109915 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:27.109939 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:27.111095 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:27.111132 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:27.111158 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:27.112630 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:27.112825 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:27.113013 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:27.118083 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:27.118067 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:27.118172 3645 server_base.cc:1034] running on GCE node
W20260927 02:42:27.118063 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:27.118511 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:27.118778 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:27.119926 3645 hybrid_clock.cc:648] HybridClock initialized: now 1790476947119918 us; error 37 us; skew 500 ppm
Sep 27 02:42:27 dist-test-slave-44kr krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:27.122445 3645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:27.122886 3645 webserver.cc:460] Webserver started at http://127.0.98.2:43421/ using document root <none> and password file <none>
I20260927 02:42:27.123064 3645 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:27.123103 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:27.123194 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:27.124197 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "bc572d3568154a3d947e180b2230c0c2"
format_stamp: "Formatted at 2026-09-27 02:42:27 on dist-test-slave-44kr"
server_key: "d028942c7a976b6736d9b03ceb535379"
server_key_iv: "fde5994e4f3de54e1a703e534f5ef974"
server_key_version: "encryptionkey@0"
I20260927 02:42:27.124547 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bc572d3568154a3d947e180b2230c0c2"
format_stamp: "Formatted at 2026-09-27 02:42:27 on dist-test-slave-44kr"
server_key: "d028942c7a976b6736d9b03ceb535379"
server_key_iv: "fde5994e4f3de54e1a703e534f5ef974"
server_key_version: "encryptionkey@0"
I20260927 02:42:27.125847 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:27.126718 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:27.126931 3645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:27.127023 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "bc572d3568154a3d947e180b2230c0c2"
format_stamp: "Formatted at 2026-09-27 02:42:27 on dist-test-slave-44kr"
server_key: "d028942c7a976b6736d9b03ceb535379"
server_key_iv: "fde5994e4f3de54e1a703e534f5ef974"
server_key_version: "encryptionkey@0"
I20260927 02:42:27.127110 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:27.150612 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:27.153016 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:27.153141 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:27.153419 3645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:27.153745 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:27.153780 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:27.153810 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:27.153827 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:27.159523 3645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38391
I20260927 02:42:27.159601 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38391 every 8 connection(s)
I20260927 02:42:27.159915 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:27.163074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 3645
I20260927 02:42:27.163182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:27.163354 392 external_mini_cluster.cc:1428] Setting key fa02be0650bd414d1cf39a16c1797953
Sep 27 02:42:27 dist-test-slave-44kr krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:27.170145 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:27.160630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39033 (local address 127.0.98.2:59565)
0927 02:42:27.160817 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:27.160857 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:27.161410 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:27.161702 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:27.161705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:27.162270 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:27.162464 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:27.162469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.163284 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.163286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:27.164326 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:27.164329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.164863 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.164864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:27.164973 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:27.165597 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:27.165608 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:27.166918 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:27.168763 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:27.168768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:27.168770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:27.168992 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:27.169267 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:27.169268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:27.169268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:27.169385 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:27.169648 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:27.169652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:27.169771 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 02:42:27.169850 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:42:27.169971 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260927 02:42:27.170091 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:27.160696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59565 (local address 127.0.98.62:39033)
0927 02:42:27.160784 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:27.160786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:27.160941 (+ 155us) server_negotiation.cc:407] Connection header received
0927 02:42:27.161517 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:27.161518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:27.161535 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:27.161618 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:27.162580 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.163181 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.164470 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.164762 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.164830 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:27.167023 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:27.167025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:27.167025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:27.167034 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:27.168658 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:27.169100 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:27.169110 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:27.169110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:27.169180 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:27.169448 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:27.169450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:27.169450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:27.169588 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:27.169657 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:27.169886 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 02:42:27.169995 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:42:27.170996 3774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39033
I20260927 02:42:27.171092 3774 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:27.171265 3774 heartbeater.cc:511] Master 127.0.98.62:39033 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:42:27.171654 3449 ts_manager.cc:194] Registered new tserver with Master: bc572d3568154a3d947e180b2230c0c2 (127.0.98.2:38391)
I20260927 02:42:27.172046 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:27.193490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39033
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:27.280704 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:27.280923 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:27.280947 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:27.282130 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:27.282181 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:27.282208 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:27.283653 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:27.283890 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:27.284106 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:27.288960 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:27.289063 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:27.288978 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:27.289084 3779 server_base.cc:1034] running on GCE node
I20260927 02:42:27.289387 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:27.289628 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:27.290791 3779 hybrid_clock.cc:648] HybridClock initialized: now 1790476947290777 us; error 29 us; skew 500 ppm
Sep 27 02:42:27 dist-test-slave-44kr krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:27.293584 3779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:27.294104 3779 webserver.cc:460] Webserver started at http://127.0.98.3:38177/ using document root <none> and password file <none>
I20260927 02:42:27.294314 3779 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:27.294370 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:27.294494 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:27.295584 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "75d7651f573f4315827515b1f0459636"
format_stamp: "Formatted at 2026-09-27 02:42:27 on dist-test-slave-44kr"
server_key: "ff76af7d00fa5a0eada7a216ba547baf"
server_key_iv: "beb481c5df531801fab924ab4265fec7"
server_key_version: "encryptionkey@0"
I20260927 02:42:27.295910 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "75d7651f573f4315827515b1f0459636"
format_stamp: "Formatted at 2026-09-27 02:42:27 on dist-test-slave-44kr"
server_key: "ff76af7d00fa5a0eada7a216ba547baf"
server_key_iv: "beb481c5df531801fab924ab4265fec7"
server_key_version: "encryptionkey@0"
I20260927 02:42:27.297287 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:27.298022 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:27.298202 3779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:42:27.298285 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "75d7651f573f4315827515b1f0459636"
format_stamp: "Formatted at 2026-09-27 02:42:27 on dist-test-slave-44kr"
server_key: "ff76af7d00fa5a0eada7a216ba547baf"
server_key_iv: "beb481c5df531801fab924ab4265fec7"
server_key_version: "encryptionkey@0"
I20260927 02:42:27.298345 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:27.309547 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:27.311833 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:27.311985 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:27.312211 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:27.312599 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:27.312634 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:27.312664 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:27.312680 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:27.318406 3779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39663
I20260927 02:42:27.318459 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39663 every 8 connection(s)
I20260927 02:42:27.318766 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:42:27 dist-test-slave-44kr krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:27.328082 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:27.319483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58633 (local address 127.0.98.62:39033)
0927 02:42:27.319558 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:27.319560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:27.319766 (+ 206us) server_negotiation.cc:407] Connection header received
0927 02:42:27.320247 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:27.320249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:27.320260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:27.320319 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:27.321268 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.321773 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.322782 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.322973 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.323036 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:27.325154 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:27.325156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:27.325156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:27.325164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:27.326637 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:27.327098 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:27.327099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:27.327099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:27.327160 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:27.327430 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:27.327431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:27.327431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:27.327595 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:27.327668 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:27.327923 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 02:42:27.327983 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:42:27.328227 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:27.319412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39033 (local address 127.0.98.3:58633)
0927 02:42:27.319626 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:27.319669 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:27.320142 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:27.320380 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:27.320413 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:27.320979 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:27.321178 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:27.321182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.321840 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.321841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:27.322678 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:27.322680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.323041 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.323042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:27.323153 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:27.323649 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:27.323657 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:27.325046 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:27.326724 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:27.326732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:27.326735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:27.327017 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:27.327237 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:27.327238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:27.327239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:27.327355 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:27.327673 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:27.327677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:27.327805 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 02:42:27.327903 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 02:42:27.328029 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260927 02:42:27.328576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 3779
I20260927 02:42:27.328677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:27.328742 3908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39033
I20260927 02:42:27.328833 3908 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:27.328869 392 external_mini_cluster.cc:1428] Setting key d55c85572ad07024878d883c907e5185
I20260927 02:42:27.329015 3908 heartbeater.cc:511] Master 127.0.98.62:39033 requested a full tablet report, sending...
I20260927 02:42:27.329355 3449 ts_manager.cc:194] Registered new tserver with Master: 75d7651f573f4315827515b1f0459636 (127.0.98.3:39663)
I20260927 02:42:27.329712 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58633
I20260927 02:42:27.330279 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:42:27 dist-test-slave-44kr krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476947, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 02:42:27 dist-test-slave-44kr krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476947, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:27.352269 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:27.344836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.0.98.62:39033)
0927 02:42:27.344890 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:27.344892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:27.344937 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:42:27.344990 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:27.344991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:27.345000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:27.345070 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:27.345685 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.346311 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.347126 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:27.347306 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:27.347360 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:27.349628 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:27.349630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:27.349630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:27.349638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:27.350976 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:27.351357 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:27.351358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:27.351358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:27.351418 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:27.351642 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:27.351643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:27.351643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:27.351824 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:27.351904 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:27.352078 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 02:42:27.352151 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 02:42:27.352546 3449 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57162
I20260927 02:42:27.353060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 3511
I20260927 02:42:27.357262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 3645
I20260927 02:42:27.360742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 3779
I20260927 02:42:27.364102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 3417
2026-09-27T02:42:27Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1763 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:27 dist-test-slave-44kr krb5kdc[3920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:27 dist-test-slave-44kr krb5kdc[3920](info): set up 2 sockets
Sep 27 02:42:27 dist-test-slave-44kr krb5kdc[3920](info): commencing operation
krb5kdc: starting...
W20260927 02:42:28.336210 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:28 dist-test-slave-44kr krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:28.464339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:28.544323 3933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:28.544651 3933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:28.544690 3933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:28.546260 3933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:28.546331 3933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:28.546355 3933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:28.546371 3933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:28.546386 3933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:28.548305 3933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:28.548619 3933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:28.548880 3933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:28.554193 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:28.554356 3933 server_base.cc:1034] running on GCE node
W20260927 02:42:28.554232 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:28.554499 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:28.554785 3933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:28.555078 3933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:28.556237 3933 hybrid_clock.cc:648] HybridClock initialized: now 1790476948556205 us; error 45 us; skew 500 ppm
Sep 27 02:42:28 dist-test-slave-44kr krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:28.558830 3933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:28.559362 3933 webserver.cc:460] Webserver started at http://127.0.98.62:44605/ using document root <none> and password file <none>
I20260927 02:42:28.559597 3933 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:28.559648 3933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:28.559775 3933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:28.560946 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "4944d164c2424edd801683de702a7d77"
format_stamp: "Formatted at 2026-09-27 02:42:28 on dist-test-slave-44kr"
server_key: "85982c9256974f5151b804dbb7586e55"
server_key_iv: "733203e149b5716f61321a60d85134c8"
server_key_version: "encryptionkey@0"
I20260927 02:42:28.561285 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "4944d164c2424edd801683de702a7d77"
format_stamp: "Formatted at 2026-09-27 02:42:28 on dist-test-slave-44kr"
server_key: "85982c9256974f5151b804dbb7586e55"
server_key_iv: "733203e149b5716f61321a60d85134c8"
server_key_version: "encryptionkey@0"
I20260927 02:42:28.562599 3933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:42:28.563453 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:28.563752 3933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:28.563849 3933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "4944d164c2424edd801683de702a7d77"
format_stamp: "Formatted at 2026-09-27 02:42:28 on dist-test-slave-44kr"
server_key: "85982c9256974f5151b804dbb7586e55"
server_key_iv: "733203e149b5716f61321a60d85134c8"
server_key_version: "encryptionkey@0"
I20260927 02:42:28.563916 3933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:28.580027 3933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:28.582419 3933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:28.582582 3933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:28.586480 3933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45065
I20260927 02:42:28.586529 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45065 every 8 connection(s)
I20260927 02:42:28.586884 3933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:28.587751 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:28.589526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 3933
I20260927 02:42:28.589625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:28.589798 392 external_mini_cluster.cc:1428] Setting key afb206b87cbd657b7b922ef19d72447f
I20260927 02:42:28.590359 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77: Bootstrap starting.
I20260927 02:42:28.591073 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:28.591444 4001 log.cc:824] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:28.592156 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77: No bootstrap required, opened a new log
I20260927 02:42:28.594368 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4944d164c2424edd801683de702a7d77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45065 } }
I20260927 02:42:28.594544 4001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:28.594578 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4944d164c2424edd801683de702a7d77, State: Initialized, Role: FOLLOWER
I20260927 02:42:28.594717 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4944d164c2424edd801683de702a7d77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45065 } }
I20260927 02:42:28.594811 4001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:28.594853 4001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:28.594940 4001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:28.595597 4001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4944d164c2424edd801683de702a7d77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45065 } }
I20260927 02:42:28.595685 4001 leader_election.cc:302] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4944d164c2424edd801683de702a7d77; no voters:
I20260927 02:42:28.595825 4001 leader_election.cc:288] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:28.595881 4006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:28.596091 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:28.596101 4006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [term 1 LEADER]: Becoming Leader. State: Replica: 4944d164c2424edd801683de702a7d77, State: Running, Role: LEADER
I20260927 02:42:28.596335 4006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4944d164c2424edd801683de702a7d77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45065 } }
I20260927 02:42:28.596729 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4944d164c2424edd801683de702a7d77. Latest consensus state: current_term: 1 leader_uuid: "4944d164c2424edd801683de702a7d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4944d164c2424edd801683de702a7d77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45065 } } }
I20260927 02:42:28.596736 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4944d164c2424edd801683de702a7d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4944d164c2424edd801683de702a7d77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45065 } } }
I20260927 02:42:28.596827 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:28.596998 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:28.597018 4015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 02:42:28 dist-test-slave-44kr krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:28.597563 4015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:28.599256 4015 catalog_manager.cc:1347] Generated new cluster ID: 2504c52fae5e4e349b19fdfc1b9b370c
I20260927 02:42:28.599340 4015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:28.602161 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:28.590369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.0.98.62:45065)
0927 02:42:28.590551 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:28.590561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:28.590592 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:42:28.591158 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:28.591168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:28.591241 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:28.591416 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:28.592057 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.593411 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.594198 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.594802 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.594911 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:28.597709 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:28.597714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:28.597715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:28.597728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:28.599831 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:28.600338 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:28.600342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:28.600342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:28.600634 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:28.601069 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:28.601070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:28.601070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:28.601472 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:28.601611 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:28.601796 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 02:42:28.601941 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260927 02:42:28.605029 4015 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:28.605592 4015 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:28.610183 4015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4944d164c2424edd801683de702a7d77: Generated new TSK 0
I20260927 02:42:28.610404 4015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:28.643850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45065
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:28.725378 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:28.725597 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:28.725630 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:28.726811 4027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:28.726854 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:28.726888 4027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:28.728423 4027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:28.728670 4027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:28.728880 4027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:28.733794 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:28.733879 4027 server_base.cc:1034] running on GCE node
W20260927 02:42:28.733803 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:28.733803 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:28.734329 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:28.734576 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:28.735728 4027 hybrid_clock.cc:648] HybridClock initialized: now 1790476948735705 us; error 39 us; skew 500 ppm
Sep 27 02:42:28 dist-test-slave-44kr krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:28.738332 4027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:28.738813 4027 webserver.cc:460] Webserver started at http://127.0.98.1:35115/ using document root <none> and password file <none>
I20260927 02:42:28.739028 4027 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:28.739084 4027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:28.739217 4027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:28.740243 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "62b19dc3ba024ee59d3bccf4014a14c6"
format_stamp: "Formatted at 2026-09-27 02:42:28 on dist-test-slave-44kr"
server_key: "05234e4b3c67dbf4c4b3b68387bf7d6a"
server_key_iv: "35e144c7f623d63129b046938bed887d"
server_key_version: "encryptionkey@0"
I20260927 02:42:28.740594 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "62b19dc3ba024ee59d3bccf4014a14c6"
format_stamp: "Formatted at 2026-09-27 02:42:28 on dist-test-slave-44kr"
server_key: "05234e4b3c67dbf4c4b3b68387bf7d6a"
server_key_iv: "35e144c7f623d63129b046938bed887d"
server_key_version: "encryptionkey@0"
I20260927 02:42:28.741849 4027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:28.742628 4042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:28.742812 4027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:28.742892 4027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "62b19dc3ba024ee59d3bccf4014a14c6"
format_stamp: "Formatted at 2026-09-27 02:42:28 on dist-test-slave-44kr"
server_key: "05234e4b3c67dbf4c4b3b68387bf7d6a"
server_key_iv: "35e144c7f623d63129b046938bed887d"
server_key_version: "encryptionkey@0"
I20260927 02:42:28.742957 4027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:28.769464 4027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:28.771795 4027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:28.771934 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:28.772163 4027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:28.772504 4027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:28.772540 4027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:28.772569 4027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:28.772586 4027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:28.778342 4027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35361
I20260927 02:42:28.778394 4155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35361 every 8 connection(s)
I20260927 02:42:28.778740 4027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:28.779215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 4027
I20260927 02:42:28.779315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:28.779526 392 external_mini_cluster.cc:1428] Setting key 2f096461164df1deee999ca9ad955740
Sep 27 02:42:28 dist-test-slave-44kr krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:42:28.789829 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:28.779553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37549 (local address 127.0.98.62:45065)
0927 02:42:28.779614 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:28.779617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:28.779877 (+ 260us) server_negotiation.cc:407] Connection header received
0927 02:42:28.781076 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:28.781078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:28.781089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:28.781133 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:28.781972 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.782750 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.783946 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.784256 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.784338 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:28.786522 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:28.786524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:28.786524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:28.786533 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:28.787973 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:28.788562 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:28.788563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:28.788563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:28.788625 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:28.789000 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:28.789001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:28.789001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:28.789197 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:28.789258 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:28.789678 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 02:42:28.789762 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:42:28.789921 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:28.779505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45065 (local address 127.0.98.1:37549)
0927 02:42:28.779742 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:28.779788 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:28.780949 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:28.781244 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:28.781249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:28.781609 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:28.781873 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:28.781879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.782855 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.782856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:28.783847 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:28.783850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.784349 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.784350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:28.784510 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:28.785025 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:28.785040 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:28.786414 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:28.788069 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:28.788077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:28.788079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:28.788479 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:28.788750 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:28.788751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:28.788751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:28.788885 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:28.789266 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:28.789274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:28.789462 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 02:42:28.789557 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:42:28.789703 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:42:28.790445 4156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45065
I20260927 02:42:28.790552 4156 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:28.790732 4156 heartbeater.cc:511] Master 127.0.98.62:45065 requested a full tablet report, sending...
I20260927 02:42:28.791290 3965 ts_manager.cc:194] Registered new tserver with Master: 62b19dc3ba024ee59d3bccf4014a14c6 (127.0.98.1:35361)
I20260927 02:42:28.792101 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:28.808176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45065
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:28.891162 4161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:28.891386 4161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:28.891415 4161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:28.892727 4161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:28.892779 4161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:28.892807 4161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:28.894338 4161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:28.894606 4161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:28.894860 4161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:28.900135 4169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:28.900166 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:28.900195 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:28.900285 4161 server_base.cc:1034] running on GCE node
I20260927 02:42:28.900528 4161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:28.900779 4161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:28.901949 4161 hybrid_clock.cc:648] HybridClock initialized: now 1790476948901935 us; error 30 us; skew 500 ppm
Sep 27 02:42:28 dist-test-slave-44kr krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:28.904647 4161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:28.905102 4161 webserver.cc:460] Webserver started at http://127.0.98.2:42231/ using document root <none> and password file <none>
I20260927 02:42:28.905324 4161 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:28.905373 4161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:28.905499 4161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:28.906540 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "08f6d99941624b908ecd9881185aded6"
format_stamp: "Formatted at 2026-09-27 02:42:28 on dist-test-slave-44kr"
server_key: "a5f832817e288778919ee19698fcfbfd"
server_key_iv: "495706bfd3a0f4ef0e6edc273d1ae6e7"
server_key_version: "encryptionkey@0"
I20260927 02:42:28.906883 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "08f6d99941624b908ecd9881185aded6"
format_stamp: "Formatted at 2026-09-27 02:42:28 on dist-test-slave-44kr"
server_key: "a5f832817e288778919ee19698fcfbfd"
server_key_iv: "495706bfd3a0f4ef0e6edc273d1ae6e7"
server_key_version: "encryptionkey@0"
I20260927 02:42:28.908166 4161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:42:28.909176 4176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:28.909433 4161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:28.909535 4161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "08f6d99941624b908ecd9881185aded6"
format_stamp: "Formatted at 2026-09-27 02:42:28 on dist-test-slave-44kr"
server_key: "a5f832817e288778919ee19698fcfbfd"
server_key_iv: "495706bfd3a0f4ef0e6edc273d1ae6e7"
server_key_version: "encryptionkey@0"
I20260927 02:42:28.909631 4161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:28.925205 4161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:28.927404 4161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:28.927544 4161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:28.927763 4161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:28.928057 4161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:28.928090 4161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:28.928119 4161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:28.928143 4161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:28.934013 4161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37407
I20260927 02:42:28.934062 4289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37407 every 8 connection(s)
I20260927 02:42:28.934376 4161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:42:28 dist-test-slave-44kr krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:28.943346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 4161
I20260927 02:42:28.943461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:28.943676 392 external_mini_cluster.cc:1428] Setting key 8fd218ab5402ad52bbb4cbbcb2d6d1d7
I20260927 02:42:28.944109 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:28.935070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41241 (local address 127.0.98.62:45065)
0927 02:42:28.935125 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:28.935136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:28.935352 (+ 216us) server_negotiation.cc:407] Connection header received
0927 02:42:28.936244 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:28.936245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:28.936257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:28.936319 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:28.936988 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.937543 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.938599 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.938796 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.938860 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:28.941184 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:28.941186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:28.941186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:28.941236 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:28.942596 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:28.943017 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:28.943018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:28.943019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:28.943078 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:28.943412 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:28.943413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:28.943413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:28.943547 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:28.943598 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:28.943952 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 02:42:28.944030 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:42:28.944300 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:28.935041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45065 (local address 127.0.98.2:41241)
0927 02:42:28.935197 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:28.935241 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:28.936124 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:28.936378 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:28.936408 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:28.936686 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:28.936883 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:28.936888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.937633 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.937634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:28.938506 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:28.938509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:28.938913 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:28.938914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:28.939103 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:28.939692 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:28.939701 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:28.941090 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:28.942691 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:28.942706 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:28.942708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:28.942945 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:28.943169 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:28.943171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:28.943172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:28.943338 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:28.943598 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:28.943604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:28.943788 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 02:42:28.943897 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 02:42:28.944074 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260927 02:42:28.944887 4290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45065
I20260927 02:42:28.944985 4290 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:28.945174 4290 heartbeater.cc:511] Master 127.0.98.62:45065 requested a full tablet report, sending...
I20260927 02:42:28.945580 3965 ts_manager.cc:194] Registered new tserver with Master: 08f6d99941624b908ecd9881185aded6 (127.0.98.2:37407)
I20260927 02:42:28.945989 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41241
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:28.972790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45065
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:29.052280 4295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:29.052542 4295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:29.052579 4295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:29.053753 4295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:29.053797 4295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:29.053831 4295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:29.055332 4295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:29.055575 4295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:29.055786 4295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:29.060707 4301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:29.060671 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:29.060797 4295 server_base.cc:1034] running on GCE node
W20260927 02:42:29.060727 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:29.061053 4295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:29.061287 4295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:29.062439 4295 hybrid_clock.cc:648] HybridClock initialized: now 1790476949062428 us; error 29 us; skew 500 ppm
Sep 27 02:42:29 dist-test-slave-44kr krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:29.065008 4295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:29.065433 4295 webserver.cc:460] Webserver started at http://127.0.98.3:37035/ using document root <none> and password file <none>
I20260927 02:42:29.065630 4295 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:29.065671 4295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:29.065762 4295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:29.066825 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "5764fac5290740ac8f30d1a8ce3ae092"
format_stamp: "Formatted at 2026-09-27 02:42:29 on dist-test-slave-44kr"
server_key: "5c7a8bb5a5b7928cbb68485ca3e75ec7"
server_key_iv: "e56c2e0f8f911f092708642eb4798f06"
server_key_version: "encryptionkey@0"
I20260927 02:42:29.067201 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5764fac5290740ac8f30d1a8ce3ae092"
format_stamp: "Formatted at 2026-09-27 02:42:29 on dist-test-slave-44kr"
server_key: "5c7a8bb5a5b7928cbb68485ca3e75ec7"
server_key_iv: "e56c2e0f8f911f092708642eb4798f06"
server_key_version: "encryptionkey@0"
I20260927 02:42:29.068578 4295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:42:29.069410 4310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:29.069567 4295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:29.069680 4295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "5764fac5290740ac8f30d1a8ce3ae092"
format_stamp: "Formatted at 2026-09-27 02:42:29 on dist-test-slave-44kr"
server_key: "5c7a8bb5a5b7928cbb68485ca3e75ec7"
server_key_iv: "e56c2e0f8f911f092708642eb4798f06"
server_key_version: "encryptionkey@0"
I20260927 02:42:29.069751 4295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:29.113593 4295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:29.115696 4295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:29.115821 4295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:29.116055 4295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:29.116349 4295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:29.116415 4295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:29.116448 4295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:29.116466 4295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:29.122005 4295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44183
I20260927 02:42:29.122089 4423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44183 every 8 connection(s)
I20260927 02:42:29.122359 4295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:29.127895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 4295
I20260927 02:42:29.128001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:29.128173 392 external_mini_cluster.cc:1428] Setting key 7650a19f8f9db8a69142627689cd74ed
Sep 27 02:42:29 dist-test-slave-44kr krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:29.132535 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:29.123127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39787 (local address 127.0.98.62:45065)
0927 02:42:29.123182 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:29.123184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:29.123366 (+ 182us) server_negotiation.cc:407] Connection header received
0927 02:42:29.124420 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:29.124421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:29.124435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:29.124505 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:29.125144 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:29.125651 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:29.126711 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:29.126910 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:29.126972 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:29.129742 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:29.129744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:29.129744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:29.129751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:29.131122 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:29.131533 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:29.131534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:29.131534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:29.131594 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:29.131890 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:29.131891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:29.131892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:29.132012 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:29.132073 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:29.132360 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 02:42:29.132470 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:42:29.132875 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:29.123109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45065 (local address 127.0.98.3:39787)
0927 02:42:29.123252 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:29.123289 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:29.124271 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:29.124583 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:29.124586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:29.124852 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:29.125052 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:29.125057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:29.125744 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:29.125745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:29.126596 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:29.126598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:29.127006 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:29.127007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:29.127271 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:29.127769 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:29.127779 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:29.129631 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:29.131210 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:29.131216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:29.131219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:29.131441 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:29.131690 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:29.131691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:29.131691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:29.131814 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:29.132093 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:29.132099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:29.132238 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 02:42:29.132331 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:42:29.132510 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260927 02:42:29.133349 4424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45065
I20260927 02:42:29.133445 4424 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:29.133632 4424 heartbeater.cc:511] Master 127.0.98.62:45065 requested a full tablet report, sending...
I20260927 02:42:29.134227 3965 ts_manager.cc:194] Registered new tserver with Master: 5764fac5290740ac8f30d1a8ce3ae092 (127.0.98.3:44183)
I20260927 02:42:29.134622 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39787
I20260927 02:42:29.140110 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:42:29 dist-test-slave-44kr krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:42:29 dist-test-slave-44kr krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:29.163486 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:29.155080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.0.98.1:35361)
0927 02:42:29.155231 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:29.155236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:29.155247 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:29.155272 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:29.155274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:29.155287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:29.155352 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:29.155960 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:29.156814 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:29.157691 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:29.158185 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:29.158252 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:29.160524 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:29.160531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:29.160531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:29.160538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:29.162031 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:29.162478 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:29.162483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:29.162483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:29.162609 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:29.162904 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:29.162905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:29.162905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:29.163095 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:29.163255 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:29.163324 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:42:29.163400 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
Sep 27 02:42:29 dist-test-slave-44kr krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:29.171654 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:29.164778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.0.98.62:45065)
0927 02:42:29.164859 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:29.164861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:29.164871 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:29.164896 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:29.164897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:29.164907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:29.164964 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:29.165460 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:29.165978 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:29.166745 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:29.166955 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:29.167011 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:29.169316 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:29.169317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:29.169317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:29.169324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:29.170483 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:29.170852 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:29.170853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:29.170853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:29.170910 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:29.171191 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:29.171193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:29.171193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:29.171350 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:29.171438 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:29.171514 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 02:42:29.171580 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 02:42:29.171901 3965 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55184
I20260927 02:42:29.172271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 4027
I20260927 02:42:29.176208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 4161
I20260927 02:42:29.179796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 4295
I20260927 02:42:29.183260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 3933
2026-09-27T02:42:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1819 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:29 dist-test-slave-44kr krb5kdc[4436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:29 dist-test-slave-44kr krb5kdc[4436](info): set up 2 sockets
Sep 27 02:42:29 dist-test-slave-44kr krb5kdc[4436](info): commencing operation
krb5kdc: starting...
W20260927 02:42:30.132157 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:30 dist-test-slave-44kr krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:30.255458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:30.335397 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:30.335614 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:30.335637 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:30.336826 4449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:30.336874 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:30.336886 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:30.336897 4449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:30.336907 4449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:30.338449 4449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:30.338707 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:30.338922 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:30.343969 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:30.344094 4449 server_base.cc:1034] running on GCE node
W20260927 02:42:30.343909 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:30.343905 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:30.344506 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:30.344775 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:30.345907 4449 hybrid_clock.cc:648] HybridClock initialized: now 1790476950345902 us; error 31 us; skew 500 ppm
Sep 27 02:42:30 dist-test-slave-44kr krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:30.348301 4449 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:30.348688 4449 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:30.348748 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:30.348857 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:30.350004 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "71e5cdcce1424aa99340daf9d8d38894"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "f604aac53b276c691a10cd94c6da7a79"
server_key_iv: "dfa17102387983ea35da7db33f27c654"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.350348 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "71e5cdcce1424aa99340daf9d8d38894"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "f604aac53b276c691a10cd94c6da7a79"
server_key_iv: "dfa17102387983ea35da7db33f27c654"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.351678 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:30.352586 4463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.352873 4449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:30.352963 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "71e5cdcce1424aa99340daf9d8d38894"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "f604aac53b276c691a10cd94c6da7a79"
server_key_iv: "dfa17102387983ea35da7db33f27c654"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.353053 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:30.361353 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:30.363919 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:30.364048 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:30.367871 4449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39927
I20260927 02:42:30.367894 4515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39927 every 8 connection(s)
I20260927 02:42:30.368175 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:30.369156 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:30.370246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 4449
I20260927 02:42:30.370330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:30.370496 392 external_mini_cluster.cc:1428] Setting key dc2e80ef110d4643303ae7beecf05053
I20260927 02:42:30.372020 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894: Bootstrap starting.
I20260927 02:42:30.372771 4516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:30.373107 4516 log.cc:824] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:30.373852 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894: No bootstrap required, opened a new log
I20260927 02:42:30.376005 4516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5cdcce1424aa99340daf9d8d38894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } }
I20260927 02:42:30.376187 4516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:30.376230 4516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e5cdcce1424aa99340daf9d8d38894, State: Initialized, Role: FOLLOWER
I20260927 02:42:30.376356 4516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5cdcce1424aa99340daf9d8d38894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } }
I20260927 02:42:30.376468 4516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:30.376523 4516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:30.376575 4516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:30.377128 4516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5cdcce1424aa99340daf9d8d38894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } }
I20260927 02:42:30.377234 4516 leader_election.cc:302] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71e5cdcce1424aa99340daf9d8d38894; no voters:
I20260927 02:42:30.377398 4516 leader_election.cc:288] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:30.377447 4521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:30.377599 4521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [term 1 LEADER]: Becoming Leader. State: Replica: 71e5cdcce1424aa99340daf9d8d38894, State: Running, Role: LEADER
Sep 27 02:42:30 dist-test-slave-44kr krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:30.377660 4516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:30.377697 4521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5cdcce1424aa99340daf9d8d38894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } }
I20260927 02:42:30.378057 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e5cdcce1424aa99340daf9d8d38894. Latest consensus state: current_term: 1 leader_uuid: "71e5cdcce1424aa99340daf9d8d38894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5cdcce1424aa99340daf9d8d38894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } } }
I20260927 02:42:30.378135 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:30.378067 4522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71e5cdcce1424aa99340daf9d8d38894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5cdcce1424aa99340daf9d8d38894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } } }
I20260927 02:42:30.378357 4522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:30.378607 4527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:30.379537 4527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:30.381170 4527 catalog_manager.cc:1347] Generated new cluster ID: 6bacbca8862d48d6aff7999ec8670e46
I20260927 02:42:30.381222 4527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:30.381927 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.371061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.0.98.62:39927)
0927 02:42:30.371254 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.371264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.371301 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:42:30.371808 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.371817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.371879 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.372065 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:30.372799 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.374111 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.374871 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.375515 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.375645 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.378211 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:30.378216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:30.378216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:30.378229 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:30.380203 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.380623 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.380626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.380626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.380729 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.380991 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.380992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.380993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.381306 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:30.381438 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.381602 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.381734 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260927 02:42:30.392972 4527 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:30.393729 4527 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:30.399961 4527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894: Generated new TSK 0
I20260927 02:42:30.400209 4527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:30.435379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39927
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:30.516084 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:30.516314 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:30.516348 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:30.517546 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:30.517590 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:30.517627 4542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:30.519121 4542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:30.519351 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:30.519562 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:42:30.524431 4542 server_base.cc:1034] running on GCE node
W20260927 02:42:30.524451 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:30.524371 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:30.524446 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:30.524799 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:30.525029 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:30.526193 4542 hybrid_clock.cc:648] HybridClock initialized: now 1790476950526162 us; error 48 us; skew 500 ppm
Sep 27 02:42:30 dist-test-slave-44kr krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:30.533226 4542 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:30.533566 4542 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:30.533617 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:30.533751 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:30.534821 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "55c58b14676a45ec840031f8225d6a9b"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "b47d157a243395a0397847f663e8e61f"
server_key_iv: "c85b52bd890fc4a02ef333f8311eacd7"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.535159 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "55c58b14676a45ec840031f8225d6a9b"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "b47d157a243395a0397847f663e8e61f"
server_key_iv: "c85b52bd890fc4a02ef333f8311eacd7"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.536357 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:30.537118 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.537285 4542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:30.537366 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "55c58b14676a45ec840031f8225d6a9b"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "b47d157a243395a0397847f663e8e61f"
server_key_iv: "c85b52bd890fc4a02ef333f8311eacd7"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.537425 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:30.572284 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:30.574513 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:30.574641 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:30.574870 4542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:30.575178 4542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:30.575212 4542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.575233 4542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:30.575255 4542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.580902 4542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39859
I20260927 02:42:30.580973 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39859 every 8 connection(s)
I20260927 02:42:30.581230 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:42:30 dist-test-slave-44kr krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:30.590400 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.581921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57649 (local address 127.0.98.62:39927)
0927 02:42:30.581984 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.581986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.582217 (+ 231us) server_negotiation.cc:407] Connection header received
0927 02:42:30.582691 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.582693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.582703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.582742 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:30.583648 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.584147 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.585265 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.585476 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.585541 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.587654 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:30.587655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:30.587656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:30.587662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:30.588954 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.589426 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.589427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.589428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.589494 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.589759 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.589760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.589760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.589889 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:30.589959 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.590235 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.590325 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:42:30.590567 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.581897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39927 (local address 127.0.98.1:57649)
0927 02:42:30.582058 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.582136 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:30.582598 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:30.582800 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:30.582804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:30.583065 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:30.583561 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.583567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.584233 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.584234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.585163 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.585166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.585563 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.585564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.585681 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.586210 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:30.586217 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:30.587554 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:30.589066 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:30.589072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:30.589074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:30.589334 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:30.589554 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:30.589555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:30.589555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:30.589661 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:30.589954 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:30.589958 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:30.590093 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 02:42:30.590180 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:42:30.590351 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260927 02:42:30.590930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 4542
I20260927 02:42:30.591024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:30.591123 4670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39927
I20260927 02:42:30.591229 4670 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:30.591243 392 external_mini_cluster.cc:1428] Setting key 9e573f500e19bf8a13526ddc49c2cc35
I20260927 02:42:30.591389 4670 heartbeater.cc:511] Master 127.0.98.62:39927 requested a full tablet report, sending...
I20260927 02:42:30.592031 4480 ts_manager.cc:194] Registered new tserver with Master: 55c58b14676a45ec840031f8225d6a9b (127.0.98.1:39859)
I20260927 02:42:30.592736 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57649
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:30.616417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39927
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:30.696800 4675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:30.697013 4675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:30.697038 4675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:30.698197 4675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:30.698244 4675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:30.698269 4675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:30.700014 4675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:30.700264 4675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:30.700511 4675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:30.705242 4681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:30.705349 4675 server_base.cc:1034] running on GCE node
W20260927 02:42:30.705260 4680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:30.705435 4683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:30.705642 4675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:30.705888 4675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:30.707036 4675 hybrid_clock.cc:648] HybridClock initialized: now 1790476950707017 us; error 29 us; skew 500 ppm
Sep 27 02:42:30 dist-test-slave-44kr krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:30.709558 4675 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:30.709887 4675 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:30.709937 4675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:30.710065 4675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:30.711187 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "8751972bc8864f6d86e2e7cee1148d11"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "8ecd75f76c508980e2a7a48ff962e12f"
server_key_iv: "6b94bc9d6d647ab919c6baa4f0f05b72"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.711521 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8751972bc8864f6d86e2e7cee1148d11"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "8ecd75f76c508980e2a7a48ff962e12f"
server_key_iv: "6b94bc9d6d647ab919c6baa4f0f05b72"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.712783 4675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:30.713565 4689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.713734 4675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:42:30.713815 4675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "8751972bc8864f6d86e2e7cee1148d11"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "8ecd75f76c508980e2a7a48ff962e12f"
server_key_iv: "6b94bc9d6d647ab919c6baa4f0f05b72"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.713873 4675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:30.722929 4675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:30.724713 4675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:30.724834 4675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:30.725030 4675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:30.725327 4675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:30.725358 4675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.725384 4675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:30.725407 4675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.730649 4675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41983
I20260927 02:42:30.730712 4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41983 every 8 connection(s)
I20260927 02:42:30.730926 4675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:30.731204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 4675
I20260927 02:42:30.731313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:30.731496 392 external_mini_cluster.cc:1428] Setting key a4e75fdd467aa3aac88d8ea5d348cb05
Sep 27 02:42:30 dist-test-slave-44kr krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:42:30.741524 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.731705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35155 (local address 127.0.98.62:39927)
0927 02:42:30.731770 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.731771 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.731963 (+ 192us) server_negotiation.cc:407] Connection header received
0927 02:42:30.732616 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.732617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.732627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.732664 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:30.734036 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.734835 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.735956 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.736257 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.736330 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.738624 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:30.738626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:30.738626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:30.738634 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:30.740036 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.740500 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.740501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.740501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.740586 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.740923 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.740924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.740925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.741055 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:30.741112 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.741390 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.741458 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:42:30.741684 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.731686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39927 (local address 127.0.98.2:35155)
0927 02:42:30.731846 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.731893 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:30.732520 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:30.732743 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:30.732748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:30.733085 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:30.733786 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.733796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.734947 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.734948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.735871 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.735874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.736360 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.736361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.736527 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.737036 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:30.737045 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:30.738530 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:30.740155 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:30.740160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:30.740162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:30.740369 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:30.740658 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:30.740659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:30.740660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:30.740825 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:30.741110 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:30.741114 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:30.741273 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 02:42:30.741355 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:42:30.741480 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260927 02:42:30.742170 4803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39927
I20260927 02:42:30.742287 4803 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:42:30.742436 4803 heartbeater.cc:511] Master 127.0.98.62:39927 requested a full tablet report, sending...
I20260927 02:42:30.742825 4480 ts_manager.cc:194] Registered new tserver with Master: 8751972bc8864f6d86e2e7cee1148d11 (127.0.98.2:41983)
I20260927 02:42:30.743178 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:30.759268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39927
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:30.838994 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:30.839267 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:30.839303 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:30.840621 4808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:30.840672 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:30.840713 4808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:30.842324 4808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:30.842566 4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:30.842792 4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:30.847895 4813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:30.847936 4816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:30.847983 4808 server_base.cc:1034] running on GCE node
W20260927 02:42:30.847942 4814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:30.848371 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:30.848649 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:30.849807 4808 hybrid_clock.cc:648] HybridClock initialized: now 1790476950849791 us; error 32 us; skew 500 ppm
Sep 27 02:42:30 dist-test-slave-44kr krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:30.852249 4808 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:30.852664 4808 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:30.852720 4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:30.852838 4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:30.853857 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "1bcea979a8d74a5e8a54fa51b665b140"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "562a935dc234f397cd804acfbe4b0e00"
server_key_iv: "dba434f7109e6433e1076fb1455b6718"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.854202 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1bcea979a8d74a5e8a54fa51b665b140"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "562a935dc234f397cd804acfbe4b0e00"
server_key_iv: "dba434f7109e6433e1076fb1455b6718"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.855423 4808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:30.856149 4822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.856361 4808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:30.856462 4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "1bcea979a8d74a5e8a54fa51b665b140"
format_stamp: "Formatted at 2026-09-27 02:42:30 on dist-test-slave-44kr"
server_key: "562a935dc234f397cd804acfbe4b0e00"
server_key_iv: "dba434f7109e6433e1076fb1455b6718"
server_key_version: "encryptionkey@0"
I20260927 02:42:30.856518 4808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:30.867658 4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:30.869839 4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:30.869956 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:30.870141 4808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:30.870450 4808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:30.870486 4808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.870508 4808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:30.870520 4808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:30.875994 4808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44689
I20260927 02:42:30.876062 4935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44689 every 8 connection(s)
I20260927 02:42:30.876289 4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:42:30 dist-test-slave-44kr krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:30.883977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 4808
I20260927 02:42:30.884140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:30.884523 392 external_mini_cluster.cc:1428] Setting key 7c00b977e81ed9bde7aa60e59461242a
I20260927 02:42:30.886545 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.877010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52853 (local address 127.0.98.62:39927)
0927 02:42:30.877078 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.877081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.877331 (+ 250us) server_negotiation.cc:407] Connection header received
0927 02:42:30.877848 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.877850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.877861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.877912 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:30.878966 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.879456 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.880444 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.880693 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.880757 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.883031 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:30.883032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:30.883033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:30.883040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:30.885106 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.885595 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.885596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.885596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.885651 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.885899 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.885901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.885901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.886040 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:30.886109 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.886405 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.886476 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:42:30.886667 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.876986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39927 (local address 127.0.98.3:52853)
0927 02:42:30.877147 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.877186 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:30.877762 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:30.877979 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:30.877983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:30.878301 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:30.878872 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.878879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.879530 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.879530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.880340 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.880342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.880788 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.880790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.880906 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.881490 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:30.881499 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:30.882867 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:30.885197 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:30.885208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:30.885211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:30.885530 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:30.885715 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:30.885715 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:30.885716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:30.885822 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:30.886103 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:30.886107 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:30.886251 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 02:42:30.886343 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:42:30.886477 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260927 02:42:30.887228 4936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39927
I20260927 02:42:30.887338 4936 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:30.887495 4936 heartbeater.cc:511] Master 127.0.98.62:39927 requested a full tablet report, sending...
I20260927 02:42:30.887965 4480 ts_manager.cc:194] Registered new tserver with Master: 1bcea979a8d74a5e8a54fa51b665b140 (127.0.98.3:44689)
I20260927 02:42:30.888348 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52853
I20260927 02:42:30.896757 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:30.904313 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.898130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.0.98.62:39927)
0927 02:42:30.898197 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.898199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.898287 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:42:30.898345 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.898347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.898357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.898403 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:30.898962 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.899451 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.900308 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.900564 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.900624 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.901793 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:30.901794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:30.901795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:30.901803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:30.903248 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.903565 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.903566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.903566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.903620 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:30.903781 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:30.903781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:30.903782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:30.903929 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:30.904015 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.904136 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.904211 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:42:30.906350 4480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:30.908228 4480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:30.915351 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.911111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.0.98.1:39859)
0927 02:42:30.911275 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.911281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.911291 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:30.911330 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.911337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.911352 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.911421 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:30.911600 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.912777 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.914101 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.914815 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.914902 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.914969 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.915215 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.915262 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260927 02:42:30.915335 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.910839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39859 (local address 127.0.0.1:45688)
0927 02:42:30.910999 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.911042 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:30.911111 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:30.911435 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:30.911439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:30.911442 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:30.911520 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.911522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.912892 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.912893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.914017 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.914020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.914925 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.914926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.914994 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.915007 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:42:30.915159 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 02:42:30.915236 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":65,"thread_start_us":50,"threads_started":1}
I20260927 02:42:30.915611 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.911291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.0.98.3:44689)
0927 02:42:30.911453 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.911458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.911469 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:30.911512 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.911515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.911531 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.911776 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:30.911863 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.912773 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.914358 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.915067 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.915123 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.915173 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.915481 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.915533 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260927 02:42:30.915436 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.910842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44689 (local address 127.0.0.1:60784)
0927 02:42:30.910999 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.911042 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:30.911111 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:30.911647 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:30.911648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:30.911651 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:30.911706 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.911707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.912878 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.912879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.914265 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.914267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.915165 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.915166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.915228 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.915232 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:30.915284 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:42:30.915319 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":69,"thread_start_us":101,"threads_started":1}
I20260927 02:42:30.916566 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.910843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41983 (local address 127.0.0.1:39578)
0927 02:42:30.911214 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.911222 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:30.911272 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:30.911750 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:30.911751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:30.911753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:30.911794 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.911795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.913040 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.913040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.914263 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.914266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.916246 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.916247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.916344 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.916349 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:30.916438 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:42:30.916484 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":33,"thread_start_us":87,"threads_started":1}
I20260927 02:42:30.916780 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.911111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.0.98.2:41983)
0927 02:42:30.911520 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.911525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.911533 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:30.911579 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.911581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.911594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.911664 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:30.911864 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.912944 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.915217 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.915921 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.915998 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.916056 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.916468 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.916512 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":93,"threads_started":1}
I20260927 02:42:30.916744 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac451c598ca410199ae3066f22632f4 (DEFAULT_TABLE table=test-table [id=a94424c536a74e80a69f60751860e2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:30.916731 4870 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac451c598ca410199ae3066f22632f4 (DEFAULT_TABLE table=test-table [id=a94424c536a74e80a69f60751860e2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:30.917088 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac451c598ca410199ae3066f22632f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:30.917091 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac451c598ca410199ae3066f22632f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:30.917577 4737 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac451c598ca410199ae3066f22632f4 (DEFAULT_TABLE table=test-table [id=a94424c536a74e80a69f60751860e2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:30.917879 4737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac451c598ca410199ae3066f22632f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:30.920038 4953 tablet_bootstrap.cc:492] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: Bootstrap starting.
I20260927 02:42:30.920497 4954 tablet_bootstrap.cc:492] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b: Bootstrap starting.
I20260927 02:42:30.920889 4953 tablet_bootstrap.cc:654] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:30.921121 4954 tablet_bootstrap.cc:654] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:30.921443 4955 tablet_bootstrap.cc:492] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11: Bootstrap starting.
I20260927 02:42:30.921476 4953 log.cc:824] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:30.921648 4954 log.cc:824] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:30.922016 4955 tablet_bootstrap.cc:654] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:30.922312 4953 tablet_bootstrap.cc:492] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: No bootstrap required, opened a new log
I20260927 02:42:30.922369 4953 ts_tablet_manager.cc:1397] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:30.922497 4954 tablet_bootstrap.cc:492] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b: No bootstrap required, opened a new log
I20260927 02:42:30.922554 4955 log.cc:824] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:30.922586 4954 ts_tablet_manager.cc:1397] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:30.923298 4955 tablet_bootstrap.cc:492] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11: No bootstrap required, opened a new log
I20260927 02:42:30.923367 4955 ts_tablet_manager.cc:1397] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:30.924012 4953 raft_consensus.cc:348] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } }
I20260927 02:42:30.924202 4953 raft_consensus.cc:374] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:30.924228 4953 raft_consensus.cc:729] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcea979a8d74a5e8a54fa51b665b140, State: Initialized, Role: FOLLOWER
I20260927 02:42:30.924345 4953 consensus_queue.cc:260] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } }
I20260927 02:42:30.924628 4953 ts_tablet_manager.cc:1428] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:30.924746 4936 heartbeater.cc:503] Master 127.0.98.62:39927 was elected leader, sending a full tablet report...
I20260927 02:42:30.925040 4954 raft_consensus.cc:348] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } }
I20260927 02:42:30.925209 4954 raft_consensus.cc:374] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:30.925251 4954 raft_consensus.cc:729] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c58b14676a45ec840031f8225d6a9b, State: Initialized, Role: FOLLOWER
I20260927 02:42:30.925359 4954 consensus_queue.cc:260] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } }
I20260927 02:42:30.925308 4955 raft_consensus.cc:348] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } }
I20260927 02:42:30.925432 4955 raft_consensus.cc:374] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:30.925462 4955 raft_consensus.cc:729] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8751972bc8864f6d86e2e7cee1148d11, State: Initialized, Role: FOLLOWER
I20260927 02:42:30.925590 4955 consensus_queue.cc:260] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } }
I20260927 02:42:30.925760 4954 ts_tablet_manager.cc:1428] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 02:42:30.925844 4955 ts_tablet_manager.cc:1428] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:30.925887 4670 heartbeater.cc:503] Master 127.0.98.62:39927 was elected leader, sending a full tablet report...
I20260927 02:42:30.925942 4803 heartbeater.cc:503] Master 127.0.98.62:39927 was elected leader, sending a full tablet report...
I20260927 02:42:30.970530 4959 raft_consensus.cc:482] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:30.970719 4959 raft_consensus.cc:504] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } }
I20260927 02:42:30.971092 4959 leader_election.cc:288] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55c58b14676a45ec840031f8225d6a9b (127.0.98.1:39859), 8751972bc8864f6d86e2e7cee1148d11 (127.0.98.2:41983)
I20260927 02:42:30.973785 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.971228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39859 (local address 127.0.98.3:50329)
0927 02:42:30.971286 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.971299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:30.971364 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:30.971497 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:30.971498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:30.971499 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:30.971567 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.971568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.972170 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.972171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.972971 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.972973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.973505 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.973505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.973589 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.973593 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:30.973659 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:42:30.973700 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:42:30.973812 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.971352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50329 (local address 127.0.98.1:39859)
0927 02:42:30.971393 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.971395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.971402 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:42:30.971426 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.971427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.971436 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.971485 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:30.971629 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.972089 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.973063 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.973429 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.973493 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.973502 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.973702 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.973744 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:42:30.974093 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.971228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41983 (local address 127.0.98.3:46957)
0927 02:42:30.971456 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.971465 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:30.971521 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:30.971638 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:30.971639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:30.971640 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:30.971711 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.971713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.972358 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.972360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.973119 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:30.973121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.973802 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.973803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:30.973880 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.973884 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:30.973981 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:42:30.974018 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":63,"thread_start_us":118,"threads_started":1}
I20260927 02:42:30.974112 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.971318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46957 (local address 127.0.98.2:41983)
0927 02:42:30.971372 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.971374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.971534 (+ 160us) server_negotiation.cc:407] Connection header received
0927 02:42:30.971567 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.971568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.971580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.971628 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:30.971787 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.972282 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.973200 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.973699 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.973760 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.973769 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.974008 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.974055 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:42:30.974128 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac451c598ca410199ae3066f22632f4" candidate_uuid: "1bcea979a8d74a5e8a54fa51b665b140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c58b14676a45ec840031f8225d6a9b" is_pre_election: true
I20260927 02:42:30.974301 4624 raft_consensus.cc:2454] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcea979a8d74a5e8a54fa51b665b140 in term 0.
I20260927 02:42:30.974452 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac451c598ca410199ae3066f22632f4" candidate_uuid: "1bcea979a8d74a5e8a54fa51b665b140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8751972bc8864f6d86e2e7cee1148d11" is_pre_election: true
I20260927 02:42:30.974505 4824 leader_election.cc:302] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcea979a8d74a5e8a54fa51b665b140, 55c58b14676a45ec840031f8225d6a9b; no voters:
I20260927 02:42:30.974614 4757 raft_consensus.cc:2454] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcea979a8d74a5e8a54fa51b665b140 in term 0.
I20260927 02:42:30.974675 4959 raft_consensus.cc:2781] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:30.974735 4959 raft_consensus.cc:482] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:30.974785 4959 raft_consensus.cc:3037] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:30.975484 4959 raft_consensus.cc:504] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } }
I20260927 02:42:30.975610 4959 leader_election.cc:288] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [CANDIDATE]: Term 1 election: Requested vote from peers 55c58b14676a45ec840031f8225d6a9b (127.0.98.1:39859), 8751972bc8864f6d86e2e7cee1148d11 (127.0.98.2:41983)
I20260927 02:42:30.975787 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac451c598ca410199ae3066f22632f4" candidate_uuid: "1bcea979a8d74a5e8a54fa51b665b140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c58b14676a45ec840031f8225d6a9b"
I20260927 02:42:30.975807 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac451c598ca410199ae3066f22632f4" candidate_uuid: "1bcea979a8d74a5e8a54fa51b665b140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8751972bc8864f6d86e2e7cee1148d11"
I20260927 02:42:30.975872 4624 raft_consensus.cc:3037] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:30.975873 4757 raft_consensus.cc:3037] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:30.976606 4757 raft_consensus.cc:2454] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcea979a8d74a5e8a54fa51b665b140 in term 1.
I20260927 02:42:30.976615 4624 raft_consensus.cc:2454] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcea979a8d74a5e8a54fa51b665b140 in term 1.
I20260927 02:42:30.976818 4823 leader_election.cc:302] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcea979a8d74a5e8a54fa51b665b140, 8751972bc8864f6d86e2e7cee1148d11; no voters:
I20260927 02:42:30.976934 4959 raft_consensus.cc:2781] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:30.977118 4959 raft_consensus.cc:686] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 1 LEADER]: Becoming Leader. State: Replica: 1bcea979a8d74a5e8a54fa51b665b140, State: Running, Role: LEADER
I20260927 02:42:30.977239 4959 consensus_queue.cc:237] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } }
I20260927 02:42:30.978173 4480 catalog_manager.cc:5495] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bcea979a8d74a5e8a54fa51b665b140 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1bcea979a8d74a5e8a54fa51b665b140" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcea979a8d74a5e8a54fa51b665b140" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44689 } health_report { overall_health: HEALTHY } } }
W20260927 02:42:30.981520 4804 tablet.cc:2369] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:30.990316 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.987841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.0.98.3:44689)
0927 02:42:30.987907 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:30.987909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:30.987917 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:30.987960 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:30.987961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:30.987982 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:30.988044 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:30.988257 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.988789 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.989556 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:30.989754 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:30.989812 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:30.989979 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:30.990091 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:30.990168 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:30.990207 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 02:42:30.990230 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:42:30.992225 4757 raft_consensus.cc:1261] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [term 1 FOLLOWER]: Refusing update from remote peer 1bcea979a8d74a5e8a54fa51b665b140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:30.992257 4624 raft_consensus.cc:1261] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [term 1 FOLLOWER]: Refusing update from remote peer 1bcea979a8d74a5e8a54fa51b665b140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:30.992611 4963 consensus_queue.cc:1035] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8751972bc8864f6d86e2e7cee1148d11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:30.992789 4959 consensus_queue.cc:1035] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c58b14676a45ec840031f8225d6a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:30.996919 4970 mvcc.cc:205] Tried to move back new op lower bound from 7333793591261401088 to 7333793591203479552. Current Snapshot: MvccSnapshot[applied={T|T < 7333793591261401088}]
I20260927 02:42:30.997978 4479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:30.998101 4479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:30.999017 4479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71e5cdcce1424aa99340daf9d8d38894: Sending DeleteTablet for 3 replicas of tablet 0ac451c598ca410199ae3066f22632f4
I20260927 02:42:30.999406 4737 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac451c598ca410199ae3066f22632f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39578
I20260927 02:42:30.999420 4870 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac451c598ca410199ae3066f22632f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60784
I20260927 02:42:30.999425 4604 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac451c598ca410199ae3066f22632f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45688
I20260927 02:42:30.999676 4976 tablet_replica.cc:330] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b: stopping tablet replica
I20260927 02:42:30.999778 4976 raft_consensus.cc:2229] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:30.999888 4976 raft_consensus.cc:2258] T 0ac451c598ca410199ae3066f22632f4 P 55c58b14676a45ec840031f8225d6a9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:30.999967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 4542
I20260927 02:42:31.004541 4977 tablet_replica.cc:330] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11: stopping tablet replica
I20260927 02:42:31.004559 4978 tablet_replica.cc:330] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: stopping tablet replica
I20260927 02:42:31.004808 4977 raft_consensus.cc:2229] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 02:42:31.004913 4465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39859 (error 108)
I20260927 02:42:31.004936 4977 raft_consensus.cc:2258] T 0ac451c598ca410199ae3066f22632f4 P 8751972bc8864f6d86e2e7cee1148d11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:31.005005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 4675
I20260927 02:42:31.005852 4978 raft_consensus.cc:2229] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:31.006006 4978 raft_consensus.cc:2258] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:31.006345 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.005883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39859 (local address 127.0.0.1:45692)
0927 02:42:31.006261 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.006308 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
I20260927 02:42:31.006680 4978 ts_tablet_manager.cc:1905] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:31.008528 4978 ts_tablet_manager.cc:1918] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 02:42:31.008584 4465 catalog_manager.cc:4584] TS 55c58b14676a45ec840031f8225d6a9b (127.0.98.1:39859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ac451c598ca410199ae3066f22632f4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39859: connect: Connection refused (error 111)
I20260927 02:42:31.008736 4978 log.cc:1196] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/0ac451c598ca410199ae3066f22632f4
I20260927 02:42:31.009012 4978 ts_tablet_manager.cc:1939] T 0ac451c598ca410199ae3066f22632f4 P 1bcea979a8d74a5e8a54fa51b665b140: Deleting consensus metadata
I20260927 02:42:31.009433 4466 catalog_manager.cc:4841] TS 1bcea979a8d74a5e8a54fa51b665b140 (127.0.98.3:44689): tablet 0ac451c598ca410199ae3066f22632f4 (table test-table [id=a94424c536a74e80a69f60751860e2e4]) successfully deleted
I20260927 02:42:31.010676 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.010560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41983 (local address 127.0.0.1:39590)
0927 02:42:31.010614 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.010644 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:42:31.010682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 4808
I20260927 02:42:31.015079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 4449
2026-09-27T02:42:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1831 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T02:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:31Z Disabled control of system clock
I20260927 02:42:31.040805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42981
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:42:31.119288 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:31.119503 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:31.119522 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:31.120904 4983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:31.120960 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:31.120973 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:31.120985 4983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:31.120996 4983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:31.122344 4983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:31.122542 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:31.122777 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:31.127444 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:31.127470 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:31.127456 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:31.127569 4983 server_base.cc:1034] running on GCE node
I20260927 02:42:31.127887 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:31.128222 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:31.129382 4983 hybrid_clock.cc:648] HybridClock initialized: now 1790476951129374 us; error 33 us; skew 500 ppm
I20260927 02:42:31.130702 4983 webserver.cc:460] Webserver started at http://127.0.98.62:38429/ using document root <none> and password file <none>
I20260927 02:42:31.130923 4983 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:31.130970 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:31.131085 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:31.132170 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "83a213fafb4e49e0b1f2a136c96ce2b7"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "845afe33f6567b31f877bf7a3a333d11"
server_key_iv: "869a9c64bc95ccf4747f5fd77def1412"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.132548 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "83a213fafb4e49e0b1f2a136c96ce2b7"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "845afe33f6567b31f877bf7a3a333d11"
server_key_iv: "869a9c64bc95ccf4747f5fd77def1412"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.133776 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:31.134615 4997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.134817 4983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:31.134907 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "83a213fafb4e49e0b1f2a136c96ce2b7"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "845afe33f6567b31f877bf7a3a333d11"
server_key_iv: "869a9c64bc95ccf4747f5fd77def1412"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.134971 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:31.138371 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:31.138649 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:31.138768 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:31.142493 4983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42981
I20260927 02:42:31.142529 5049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42981 every 8 connection(s)
I20260927 02:42:31.142840 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:31.143451 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:31.144591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 4983
I20260927 02:42:31.144688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:31.144861 392 external_mini_cluster.cc:1428] Setting key ae70d419dc7c511bd25d95501019173b
I20260927 02:42:31.146924 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7: Bootstrap starting.
I20260927 02:42:31.147348 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.145408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.0.98.62:42981)
0927 02:42:31.145596 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.145604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.145629 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:42:31.146048 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.146055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.146098 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.146227 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.146904 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.146907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.146907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.146911 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.146954 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.146964 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.147032 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.147104 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.147152 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260927 02:42:31.147588 5050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:31.147873 5050 log.cc:824] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:31.148602 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7: No bootstrap required, opened a new log
I20260927 02:42:31.149992 5050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a213fafb4e49e0b1f2a136c96ce2b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42981 } }
I20260927 02:42:31.150146 5050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:31.150177 5050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a213fafb4e49e0b1f2a136c96ce2b7, State: Initialized, Role: FOLLOWER
I20260927 02:42:31.150290 5050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a213fafb4e49e0b1f2a136c96ce2b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42981 } }
I20260927 02:42:31.150374 5050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:31.150414 5050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:31.150463 5050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:31.151042 5050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a213fafb4e49e0b1f2a136c96ce2b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42981 } }
I20260927 02:42:31.151137 5050 leader_election.cc:302] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83a213fafb4e49e0b1f2a136c96ce2b7; no voters:
I20260927 02:42:31.151298 5050 leader_election.cc:288] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:31.151552 5050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:31.151716 5055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:31.151870 5055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [term 1 LEADER]: Becoming Leader. State: Replica: 83a213fafb4e49e0b1f2a136c96ce2b7, State: Running, Role: LEADER
I20260927 02:42:31.151980 5055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a213fafb4e49e0b1f2a136c96ce2b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42981 } }
I20260927 02:42:31.152369 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83a213fafb4e49e0b1f2a136c96ce2b7. Latest consensus state: current_term: 1 leader_uuid: "83a213fafb4e49e0b1f2a136c96ce2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a213fafb4e49e0b1f2a136c96ce2b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42981 } } }
I20260927 02:42:31.152486 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:31.152420 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83a213fafb4e49e0b1f2a136c96ce2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a213fafb4e49e0b1f2a136c96ce2b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42981 } } }
I20260927 02:42:31.152585 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:31.152678 5065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:31.153126 5065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:31.154747 5065 catalog_manager.cc:1347] Generated new cluster ID: b6adb8159c7b40039aa0dd233b739339
I20260927 02:42:31.154814 5065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:31.161513 5065 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:31.161674 5065 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:31.166723 5065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7: Generated new TSK 0
I20260927 02:42:31.166916 5065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:31.168556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42981
--builtin_ntp_servers=127.0.98.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:42:31.250062 5074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:31.250283 5074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:31.250303 5074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:31.251585 5074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:31.251627 5074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:31.251653 5074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:31.253150 5074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:31.253361 5074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:31.253634 5074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:31.258457 5080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:31.258556 5082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:31.258468 5079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:31.258842 5074 server_base.cc:1034] running on GCE node
I20260927 02:42:31.259039 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:31.259271 5074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:31.260440 5074 hybrid_clock.cc:648] HybridClock initialized: now 1790476951260427 us; error 28 us; skew 500 ppm
I20260927 02:42:31.261713 5074 webserver.cc:460] Webserver started at http://127.0.98.1:42175/ using document root <none> and password file <none>
I20260927 02:42:31.261930 5074 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:31.261976 5074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:31.262095 5074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:31.263163 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "e4a3becddfb147c8bbe9834d2c1e7f57"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "1b1bd3e8cc5ac211c400bc905be3b694"
server_key_iv: "1551aa70d26373379e60fb5d9ebe3973"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.263479 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e4a3becddfb147c8bbe9834d2c1e7f57"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "1b1bd3e8cc5ac211c400bc905be3b694"
server_key_iv: "1551aa70d26373379e60fb5d9ebe3973"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.264743 5074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:31.265455 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.265596 5074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:42:31.265683 5074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "e4a3becddfb147c8bbe9834d2c1e7f57"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "1b1bd3e8cc5ac211c400bc905be3b694"
server_key_iv: "1551aa70d26373379e60fb5d9ebe3973"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.265753 5074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:31.269134 5074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:31.269474 5074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:31.269596 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:31.269809 5074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:31.270129 5074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:31.270174 5074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.270203 5074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:31.270227 5074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.277206 5074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34175
I20260927 02:42:31.277271 5201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34175 every 8 connection(s)
I20260927 02:42:31.277623 5074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:31.281154 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.278490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37609 (local address 127.0.98.62:42981)
0927 02:42:31.278556 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.278558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.278876 (+ 318us) server_negotiation.cc:407] Connection header received
0927 02:42:31.279537 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.279539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.279544 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.279601 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.280933 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.280934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.280934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.280938 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.280953 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.280958 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.280997 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.281078 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.281090 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:42:31.281363 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.278476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42981 (local address 127.0.98.1:37609)
0927 02:42:31.278708 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.278764 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:31.279421 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:31.279688 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:31.279693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:31.279704 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:31.280757 (+ 1053us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:31.280758 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:31.280798 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:31.280835 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:31.281009 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:31.281013 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:31.281017 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:31.281088 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:42:31.281153 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260927 02:42:31.281951 5202 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42981
I20260927 02:42:31.282042 5202 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:31.282244 5202 heartbeater.cc:511] Master 127.0.98.62:42981 requested a full tablet report, sending...
I20260927 02:42:31.282779 5014 ts_manager.cc:194] Registered new tserver with Master: e4a3becddfb147c8bbe9834d2c1e7f57 (127.0.98.1:34175)
I20260927 02:42:31.285279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 5074
I20260927 02:42:31.285377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:31.285511 392 external_mini_cluster.cc:1428] Setting key 3131f9c2e670e83bee2a96ba71c99cbe
I20260927 02:42:31.286700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42981
--builtin_ntp_servers=127.0.98.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:42:31.366784 5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:31.367005 5205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:31.367028 5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:31.368287 5205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:31.368352 5205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:31.368408 5205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:31.369913 5205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:31.370167 5205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:31.370453 5205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:31.375383 5213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:31.375385 5210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:31.375495 5205 server_base.cc:1034] running on GCE node
W20260927 02:42:31.375394 5211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:31.375726 5205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:31.375967 5205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:31.377134 5205 hybrid_clock.cc:648] HybridClock initialized: now 1790476951377117 us; error 37 us; skew 500 ppm
I20260927 02:42:31.378435 5205 webserver.cc:460] Webserver started at http://127.0.98.2:38827/ using document root <none> and password file <none>
I20260927 02:42:31.378669 5205 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:31.378746 5205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:31.378880 5205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:31.379940 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "36831151f1b14ad28e975bb1778890ff"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "483f4f6189ef45dbc5220ab511bacf28"
server_key_iv: "f04fc4333a47eaebd8b280bbcc135856"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.380272 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "36831151f1b14ad28e975bb1778890ff"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "483f4f6189ef45dbc5220ab511bacf28"
server_key_iv: "f04fc4333a47eaebd8b280bbcc135856"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.381598 5205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:31.382397 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.382591 5205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:31.382678 5205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "36831151f1b14ad28e975bb1778890ff"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "483f4f6189ef45dbc5220ab511bacf28"
server_key_iv: "f04fc4333a47eaebd8b280bbcc135856"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.382745 5205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:31.386193 5205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:31.386472 5205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:31.386602 5205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:31.386804 5205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:31.387106 5205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:31.387140 5205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.387168 5205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:31.387187 5205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.393471 5205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34483
I20260927 02:42:31.393522 5332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34483 every 8 connection(s)
I20260927 02:42:31.393867 5205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:31.397190 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.394681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33533 (local address 127.0.98.62:42981)
0927 02:42:31.394750 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.394752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.394957 (+ 205us) server_negotiation.cc:407] Connection header received
0927 02:42:31.395536 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.395537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.395545 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.395597 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.396949 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.396950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.396951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.396955 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.396971 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.396975 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.397027 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.397100 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.397114 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:42:31.397403 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.394602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42981 (local address 127.0.98.2:33533)
0927 02:42:31.394836 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.394874 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:31.395423 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:31.395671 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:31.395675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:31.395684 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:31.396795 (+ 1111us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:31.396796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:31.396826 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:31.396858 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:31.397032 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:31.397036 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:31.397040 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:31.397108 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:42:31.397182 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":119,"threads_started":1}
I20260927 02:42:31.397964 5333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42981
I20260927 02:42:31.398047 5333 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:31.398196 5333 heartbeater.cc:511] Master 127.0.98.62:42981 requested a full tablet report, sending...
I20260927 02:42:31.398617 5014 ts_manager.cc:194] Registered new tserver with Master: 36831151f1b14ad28e975bb1778890ff (127.0.98.2:34483)
I20260927 02:42:31.401432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 5205
I20260927 02:42:31.401520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:31.401721 392 external_mini_cluster.cc:1428] Setting key 6215654ba3c56ff1ef08209f3b90e502
I20260927 02:42:31.402774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42981
--builtin_ntp_servers=127.0.98.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:42:31.482489 5336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:31.482709 5336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:31.482730 5336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:31.483899 5336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:31.483942 5336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:31.483966 5336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:31.485314 5336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:31.485541 5336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:31.485823 5336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:31.490813 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:31.490866 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:31.490965 5336 server_base.cc:1034] running on GCE node
W20260927 02:42:31.490814 5344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:31.491266 5336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:31.491533 5336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:31.492724 5336 hybrid_clock.cc:648] HybridClock initialized: now 1790476951492710 us; error 30 us; skew 500 ppm
I20260927 02:42:31.494127 5336 webserver.cc:460] Webserver started at http://127.0.98.3:38759/ using document root <none> and password file <none>
I20260927 02:42:31.494369 5336 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:31.494419 5336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:31.494546 5336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:31.495639 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "1ab4b9af227a4ab393f5259c8b56b238"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "f2ad7e405883e483d5d9a135f9635034"
server_key_iv: "38b31e62b681eafc190a7c1e02ec9b87"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.495960 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1ab4b9af227a4ab393f5259c8b56b238"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "f2ad7e405883e483d5d9a135f9635034"
server_key_iv: "38b31e62b681eafc190a7c1e02ec9b87"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.497254 5336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:31.497972 5350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.498236 5336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:31.498329 5336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "1ab4b9af227a4ab393f5259c8b56b238"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "f2ad7e405883e483d5d9a135f9635034"
server_key_iv: "38b31e62b681eafc190a7c1e02ec9b87"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.498394 5336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:31.501696 5336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:31.501951 5336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:31.502071 5336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:31.502270 5336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:31.502554 5336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:31.502584 5336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.502610 5336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:31.502635 5336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.508255 5336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40929
I20260927 02:42:31.508421 5463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40929 every 8 connection(s)
I20260927 02:42:31.508687 5336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:31.511672 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.509451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50241 (local address 127.0.98.62:42981)
0927 02:42:31.509516 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.509518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.509717 (+ 199us) server_negotiation.cc:407] Connection header received
0927 02:42:31.510193 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.510194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.510201 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.510276 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.511459 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.511460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.511460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.511465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.511481 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.511485 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.511535 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.511597 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.511610 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:42:31.511842 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.509413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42981 (local address 127.0.98.3:50241)
0927 02:42:31.509588 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.509630 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:31.510097 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:31.510337 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:31.510341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:31.510351 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:31.511304 (+ 953us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:31.511305 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:31.511337 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:31.511368 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:31.511535 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:31.511537 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:31.511540 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:31.511592 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:42:31.511643 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260927 02:42:31.512247 5464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42981
I20260927 02:42:31.512336 5464 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:31.512501 5464 heartbeater.cc:511] Master 127.0.98.62:42981 requested a full tablet report, sending...
I20260927 02:42:31.512811 5014 ts_manager.cc:194] Registered new tserver with Master: 1ab4b9af227a4ab393f5259c8b56b238 (127.0.98.3:40929)
I20260927 02:42:31.517540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 5336
I20260927 02:42:31.517623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:31.517763 392 external_mini_cluster.cc:1428] Setting key d887546a72a9cea9fff38b1fd3497a1e
I20260927 02:42:31.518550 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:31.520803 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.519814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.0.98.62:42981)
0927 02:42:31.519884 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.519885 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.519921 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:42:31.519991 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.519991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.519995 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.520031 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.520614 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.520615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.520615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.520618 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.520631 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.520634 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.520674 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.520723 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.520735 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:42:31.522451 5014 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:31.523988 5014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:31.527652 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.526117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34175 (local address 127.0.0.1:54454)
0927 02:42:31.526321 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.526362 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:31.526413 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:31.527001 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:31.527002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:31.527005 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:31.527290 (+ 285us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:31.527290 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:31.527295 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:31.527304 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:31.527521 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:31.527522 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:31.527522 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 02:42:31.527570 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:42:31.527585 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260927 02:42:31.527695 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.526263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.0.98.3:40929)
0927 02:42:31.526434 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.526439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.526598 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:42:31.526633 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.526636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.526645 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.526726 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.527331 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.527337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.527337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.527341 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.527377 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.527402 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.527453 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.527598 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.527616 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
I20260927 02:42:31.527881 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.526165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40929 (local address 127.0.0.1:40476)
0927 02:42:31.526511 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.526522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:31.526584 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:31.526734 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:31.526741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:31.526748 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:31.527209 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:31.527209 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:31.527226 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:31.527249 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:31.527462 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:31.527468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:31.527471 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:31.527814 (+ 343us) client_negotiation.cc:240] Negotiation successful
0927 02:42:31.527828 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":104,"threads_started":1}
I20260927 02:42:31.527931 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.526338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.0.98.1:34175)
0927 02:42:31.526519 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.526523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.526537 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:42:31.526565 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.526567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.526578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.526655 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.527366 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.527374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.527375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.527380 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.527417 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.527439 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.527496 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.527833 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.527849 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260927 02:42:31.528566 5398 tablet_service.cc:1467] Processing CreateTablet for tablet 3131a8ddd38a48c586e11f1a653654b0 (DEFAULT_TABLE table=test-table [id=5d196d3db635482c9407f5107bdd7158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:31.528591 5136 tablet_service.cc:1467] Processing CreateTablet for tablet 3131a8ddd38a48c586e11f1a653654b0 (DEFAULT_TABLE table=test-table [id=5d196d3db635482c9407f5107bdd7158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:31.528868 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131a8ddd38a48c586e11f1a653654b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:31.528909 5136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131a8ddd38a48c586e11f1a653654b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:31.529728 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.526321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34483 (local address 127.0.0.1:57088)
0927 02:42:31.526701 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.526710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:31.526744 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:31.527529 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:31.527530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:31.527532 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:31.529364 (+ 1832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:31.529365 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:31.529372 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:31.529384 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:31.529613 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:31.529614 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:31.529615 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:42:31.529657 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:42:31.529671 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":66,"threads_started":1}
I20260927 02:42:31.529778 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.526503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.0.98.2:34483)
0927 02:42:31.527056 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.527060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.527070 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:31.527445 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.527451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.527461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.527526 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.529459 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.529466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.529467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.529471 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.529507 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.529530 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.529614 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.529675 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.529692 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":108,"threads_started":1}
I20260927 02:42:31.531546 5478 tablet_bootstrap.cc:492] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57: Bootstrap starting.
I20260927 02:42:31.532024 5479 tablet_bootstrap.cc:492] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: Bootstrap starting.
I20260927 02:42:31.532298 5478 tablet_bootstrap.cc:654] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:31.532196 5267 tablet_service.cc:1467] Processing CreateTablet for tablet 3131a8ddd38a48c586e11f1a653654b0 (DEFAULT_TABLE table=test-table [id=5d196d3db635482c9407f5107bdd7158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:31.532541 5267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131a8ddd38a48c586e11f1a653654b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:31.532641 5479 tablet_bootstrap.cc:654] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:31.532637 5478 log.cc:824] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:31.532931 5479 log.cc:824] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:31.533335 5478 tablet_bootstrap.cc:492] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57: No bootstrap required, opened a new log
I20260927 02:42:31.533406 5478 ts_tablet_manager.cc:1397] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:31.533627 5479 tablet_bootstrap.cc:492] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: No bootstrap required, opened a new log
I20260927 02:42:31.533685 5479 ts_tablet_manager.cc:1397] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:31.535372 5482 tablet_bootstrap.cc:492] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff: Bootstrap starting.
I20260927 02:42:31.535897 5482 tablet_bootstrap.cc:654] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:31.535858 5478 raft_consensus.cc:348] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } }
I20260927 02:42:31.536034 5478 raft_consensus.cc:374] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:31.536087 5478 raft_consensus.cc:729] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a3becddfb147c8bbe9834d2c1e7f57, State: Initialized, Role: FOLLOWER
I20260927 02:42:31.536216 5478 consensus_queue.cc:260] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } }
I20260927 02:42:31.536232 5482 log.cc:824] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:31.536252 5479 raft_consensus.cc:348] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } }
I20260927 02:42:31.536419 5479 raft_consensus.cc:374] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:31.536455 5479 raft_consensus.cc:729] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab4b9af227a4ab393f5259c8b56b238, State: Initialized, Role: FOLLOWER
I20260927 02:42:31.536487 5478 ts_tablet_manager.cc:1428] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:31.536579 5479 consensus_queue.cc:260] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } }
I20260927 02:42:31.536617 5202 heartbeater.cc:503] Master 127.0.98.62:42981 was elected leader, sending a full tablet report...
I20260927 02:42:31.536829 5464 heartbeater.cc:503] Master 127.0.98.62:42981 was elected leader, sending a full tablet report...
I20260927 02:42:31.536823 5479 ts_tablet_manager.cc:1428] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:31.537484 5482 tablet_bootstrap.cc:492] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff: No bootstrap required, opened a new log
I20260927 02:42:31.537709 5482 ts_tablet_manager.cc:1397] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:42:31.539624 5482 raft_consensus.cc:348] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } }
I20260927 02:42:31.539740 5482 raft_consensus.cc:374] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:31.539768 5482 raft_consensus.cc:729] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36831151f1b14ad28e975bb1778890ff, State: Initialized, Role: FOLLOWER
I20260927 02:42:31.539880 5482 consensus_queue.cc:260] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } }
I20260927 02:42:31.540102 5482 ts_tablet_manager.cc:1428] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:42:31.540141 5333 heartbeater.cc:503] Master 127.0.98.62:42981 was elected leader, sending a full tablet report...
I20260927 02:42:31.619391 5483 raft_consensus.cc:482] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:31.619524 5483 raft_consensus.cc:504] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } }
I20260927 02:42:31.619925 5483 leader_election.cc:288] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ab4b9af227a4ab393f5259c8b56b238 (127.0.98.3:40929), 36831151f1b14ad28e975bb1778890ff (127.0.98.2:34483)
I20260927 02:42:31.621156 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.620118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34483 (local address 127.0.98.1:50775)
0927 02:42:31.620174 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.620185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:31.620281 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:31.620420 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:31.620420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:31.620422 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:31.620830 (+ 408us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:31.620831 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:31.620838 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:31.620851 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:31.621038 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:31.621038 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:31.621039 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:42:31.621074 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 02:42:31.621087 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 02:42:31.621205 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.620210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47425 (local address 127.0.98.3:40929)
0927 02:42:31.620260 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.620262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.620402 (+ 140us) server_negotiation.cc:407] Connection header received
0927 02:42:31.620452 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.620453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.620458 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.620504 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.621018 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.621019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.621019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.621023 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.621037 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.621041 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.621093 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.621139 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.621151 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:42:31.621182 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.620231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50775 (local address 127.0.98.2:34483)
0927 02:42:31.620293 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.620295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.620304 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:31.620343 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.620344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.620348 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.620414 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.620940 (+ 526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.620941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.620942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.620946 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.620971 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.620976 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.621035 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.621089 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.621103 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:42:31.621263 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.620133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40929 (local address 127.0.98.1:47425)
0927 02:42:31.620326 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.620335 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:31.620401 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:31.620528 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:31.620529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:31.620530 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:31.620931 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:31.620931 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:31.620936 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:31.620945 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:31.621096 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:31.621096 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:31.621097 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:42:31.621139 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:42:31.621149 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":117,"threads_started":1}
I20260927 02:42:31.621508 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131a8ddd38a48c586e11f1a653654b0" candidate_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36831151f1b14ad28e975bb1778890ff" is_pre_election: true
I20260927 02:42:31.621523 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131a8ddd38a48c586e11f1a653654b0" candidate_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ab4b9af227a4ab393f5259c8b56b238" is_pre_election: true
I20260927 02:42:31.621660 5418 raft_consensus.cc:2454] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4a3becddfb147c8bbe9834d2c1e7f57 in term 0.
I20260927 02:42:31.621667 5287 raft_consensus.cc:2454] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4a3becddfb147c8bbe9834d2c1e7f57 in term 0.
I20260927 02:42:31.621909 5092 leader_election.cc:302] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab4b9af227a4ab393f5259c8b56b238, e4a3becddfb147c8bbe9834d2c1e7f57; no voters:
I20260927 02:42:31.622895 5483 raft_consensus.cc:2781] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:31.622953 5483 raft_consensus.cc:482] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:31.622969 5483 raft_consensus.cc:3037] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:31.623659 5483 raft_consensus.cc:504] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } }
I20260927 02:42:31.623785 5483 leader_election.cc:288] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [CANDIDATE]: Term 1 election: Requested vote from peers 1ab4b9af227a4ab393f5259c8b56b238 (127.0.98.3:40929), 36831151f1b14ad28e975bb1778890ff (127.0.98.2:34483)
I20260927 02:42:31.623927 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131a8ddd38a48c586e11f1a653654b0" candidate_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ab4b9af227a4ab393f5259c8b56b238"
I20260927 02:42:31.623946 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131a8ddd38a48c586e11f1a653654b0" candidate_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36831151f1b14ad28e975bb1778890ff"
I20260927 02:42:31.624002 5418 raft_consensus.cc:3037] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:31.624006 5287 raft_consensus.cc:3037] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:31.624722 5287 raft_consensus.cc:2454] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a3becddfb147c8bbe9834d2c1e7f57 in term 1.
I20260927 02:42:31.624796 5418 raft_consensus.cc:2454] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a3becddfb147c8bbe9834d2c1e7f57 in term 1.
I20260927 02:42:31.624907 5090 leader_election.cc:302] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36831151f1b14ad28e975bb1778890ff, e4a3becddfb147c8bbe9834d2c1e7f57; no voters:
I20260927 02:42:31.625000 5483 raft_consensus.cc:2781] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:31.625144 5483 raft_consensus.cc:686] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 1 LEADER]: Becoming Leader. State: Replica: e4a3becddfb147c8bbe9834d2c1e7f57, State: Running, Role: LEADER
I20260927 02:42:31.625231 5483 consensus_queue.cc:237] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } }
I20260927 02:42:31.626111 5014 catalog_manager.cc:5495] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4a3becddfb147c8bbe9834d2c1e7f57 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4a3becddfb147c8bbe9834d2c1e7f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } health_report { overall_health: UNKNOWN } } }
W20260927 02:42:31.644496 5334 tablet.cc:2369] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:31.645522 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.644450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.0.98.1:34175)
0927 02:42:31.644501 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.644503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.644512 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:31.644649 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.644650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.644668 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.644724 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.645346 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.645347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.645348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.645352 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.645367 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.645371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.645428 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.645459 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.645471 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:42:31.647269 5287 raft_consensus.cc:1261] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff [term 1 FOLLOWER]: Refusing update from remote peer e4a3becddfb147c8bbe9834d2c1e7f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:31.647293 5418 raft_consensus.cc:1261] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [term 1 FOLLOWER]: Refusing update from remote peer e4a3becddfb147c8bbe9834d2c1e7f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:31.647675 5488 consensus_queue.cc:1035] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36831151f1b14ad28e975bb1778890ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:31.647830 5483 consensus_queue.cc:1035] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab4b9af227a4ab393f5259c8b56b238" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:31.650125 5491 mvcc.cc:205] Tried to move back new op lower bound from 7333793593944600576 to 7333793593857671168. Current Snapshot: MvccSnapshot[applied={T|T < 7333793593944600576}]
I20260927 02:42:31.651430 5493 mvcc.cc:205] Tried to move back new op lower bound from 7333793593944600576 to 7333793593857671168. Current Snapshot: MvccSnapshot[applied={T|T < 7333793593944600576 or (T in {7333793593944600576})}]
I20260927 02:42:31.652447 5013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:31.652535 5013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:31.653244 5013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83a213fafb4e49e0b1f2a136c96ce2b7: Sending DeleteTablet for 3 replicas of tablet 3131a8ddd38a48c586e11f1a653654b0
I20260927 02:42:31.653553 5398 tablet_service.cc:1514] Processing DeleteTablet for tablet 3131a8ddd38a48c586e11f1a653654b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:31 UTC) from {username='slave'} at 127.0.0.1:40476
I20260927 02:42:31.653736 5500 tablet_replica.cc:330] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: stopping tablet replica
I20260927 02:42:31.653883 5500 raft_consensus.cc:2229] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:31.653904 5267 tablet_service.cc:1514] Processing DeleteTablet for tablet 3131a8ddd38a48c586e11f1a653654b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:31 UTC) from {username='slave'} at 127.0.0.1:57088
I20260927 02:42:31.654001 5500 raft_consensus.cc:2258] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:31.654098 5136 tablet_service.cc:1514] Processing DeleteTablet for tablet 3131a8ddd38a48c586e11f1a653654b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:31 UTC) from {username='slave'} at 127.0.0.1:54454
I20260927 02:42:31.654295 5502 tablet_replica.cc:330] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57: stopping tablet replica
I20260927 02:42:31.654426 5502 raft_consensus.cc:2229] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:31.654436 5500 ts_tablet_manager.cc:1905] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:31.654554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 5074
I20260927 02:42:31.654579 5502 raft_consensus.cc:2258] T 3131a8ddd38a48c586e11f1a653654b0 P e4a3becddfb147c8bbe9834d2c1e7f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:31.656705 5500 ts_tablet_manager.cc:1918] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:31.656854 5500 log.cc:1196] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/3131a8ddd38a48c586e11f1a653654b0
I20260927 02:42:31.657168 5500 ts_tablet_manager.cc:1939] T 3131a8ddd38a48c586e11f1a653654b0 P 1ab4b9af227a4ab393f5259c8b56b238: Deleting consensus metadata
I20260927 02:42:31.657560 5001 catalog_manager.cc:4841] TS 1ab4b9af227a4ab393f5259c8b56b238 (127.0.98.3:40929): tablet 3131a8ddd38a48c586e11f1a653654b0 (table test-table [id=5d196d3db635482c9407f5107bdd7158]) successfully deleted
W20260927 02:42:31.659812 4999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34175 (error 108)
I20260927 02:42:31.660313 5501 tablet_replica.cc:330] T 3131a8ddd38a48c586e11f1a653654b0 P 36831151f1b14ad28e975bb1778890ff: stopping tablet replica
I20260927 02:42:31.660307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 5205
I20260927 02:42:31.660337 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.660194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34175 (local address 127.0.0.1:54480)
0927 02:42:31.660255 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.660304 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:42:31.661815 4999 catalog_manager.cc:4584] TS e4a3becddfb147c8bbe9834d2c1e7f57 (127.0.98.1:34175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3131a8ddd38a48c586e11f1a653654b0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34175: connect: Connection refused (error 111)
I20260927 02:42:31.664459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 5336
I20260927 02:42:31.664542 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.664443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34483 (local address 127.0.0.1:57094)
0927 02:42:31.664488 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.664513 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:42:31.668658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 4983
2026-09-27T02:42:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (653 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 02:42:31.687726 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:42:31.688885 392 webserver.cc:460] Webserver started at https://127.0.0.1:39469/ using document root <none> and password file <none>
I20260927 02:42:31.688989 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:42:31.689191 392 webserver.cc:460] Webserver started at http://127.0.0.1:33977/ using document root <none> and password file <none>
2026-09-27T02:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:31Z Disabled control of system clock
I20260927 02:42:31.697103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35775
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35775
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39469/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/testchainca.cert with env {}
W20260927 02:42:31.782428 5509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:31.782649 5509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:31.782672 5509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:31.783845 5509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:31.783887 5509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:31.783900 5509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:31.783910 5509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:31.783922 5509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:42:31.783933 5509 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:42:31.783943 5509 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39469/jwks.json
I20260927 02:42:31.785566 5509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39469/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:31.785794 5509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:31.785974 5509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:31.790863 5517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:31.790870 5515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:31.791255 5509 server_base.cc:1034] running on GCE node
W20260927 02:42:31.790870 5514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:31.791636 5509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:31.791970 5509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:31.793174 5509 hybrid_clock.cc:648] HybridClock initialized: now 1790476951793152 us; error 38 us; skew 500 ppm
I20260927 02:42:31.794575 5509 webserver.cc:460] Webserver started at http://127.0.98.62:38685/ using document root <none> and password file <none>
I20260927 02:42:31.794814 5509 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:31.794864 5509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:31.794986 5509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:31.796058 5509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "fba58a357c6b438b8630691660c7aa15"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "75312f944aa97696c7f19bddf3928662"
server_key_iv: "da7b9386cfddf0346109984c72d5ba5f"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.796407 5509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "fba58a357c6b438b8630691660c7aa15"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "75312f944aa97696c7f19bddf3928662"
server_key_iv: "da7b9386cfddf0346109984c72d5ba5f"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.797716 5509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:31.798496 5523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:31.798674 5509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:31.798754 5509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "fba58a357c6b438b8630691660c7aa15"
format_stamp: "Formatted at 2026-09-27 02:42:31 on dist-test-slave-44kr"
server_key: "75312f944aa97696c7f19bddf3928662"
server_key_iv: "da7b9386cfddf0346109984c72d5ba5f"
server_key_version: "encryptionkey@0"
I20260927 02:42:31.798822 5509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:31.813234 5509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:31.813560 5509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:31.813694 5509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:31.817781 5509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35775
I20260927 02:42:31.817830 5575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35775 every 8 connection(s)
I20260927 02:42:31.818166 5509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:31.819159 5576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:31.821489 5576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15: Bootstrap starting.
I20260927 02:42:31.821532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 5509
I20260927 02:42:31.821621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:31.821797 392 external_mini_cluster.cc:1428] Setting key 5f1b05be60835cbceddbb1f7d9b8ac48
I20260927 02:42:31.822121 5576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:31.822448 5576 log.cc:824] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:31.823163 5576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15: No bootstrap required, opened a new log
I20260927 02:42:31.825400 5576 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba58a357c6b438b8630691660c7aa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35775 } }
I20260927 02:42:31.825531 5576 raft_consensus.cc:374] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:31.825554 5576 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba58a357c6b438b8630691660c7aa15, State: Initialized, Role: FOLLOWER
I20260927 02:42:31.825651 5576 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba58a357c6b438b8630691660c7aa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35775 } }
I20260927 02:42:31.825721 5576 raft_consensus.cc:388] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:31.825752 5576 raft_consensus.cc:482] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:31.825788 5576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:31.827512 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.822272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.0.98.62:35775)
0927 02:42:31.822481 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.822490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.822522 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:42:31.823017 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.823026 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.823103 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.823254 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:31.823470 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:31.824829 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:31.825705 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:31.826289 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:31.826413 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:31.826950 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:31.826953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:31.826954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:31.827110 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:31.827141 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:31.827152 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:31.827202 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:31.827313 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 02:42:31.827372 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260927 02:42:31.828606 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:31.915279 5576 raft_consensus.cc:504] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba58a357c6b438b8630691660c7aa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35775 } }
I20260927 02:42:31.915457 5576 leader_election.cc:302] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fba58a357c6b438b8630691660c7aa15; no voters:
I20260927 02:42:31.915763 5576 leader_election.cc:288] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:31.915825 5581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:31.915984 5581 raft_consensus.cc:686] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [term 1 LEADER]: Becoming Leader. State: Replica: fba58a357c6b438b8630691660c7aa15, State: Running, Role: LEADER
I20260927 02:42:31.916076 5576 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:31.916097 5581 consensus_queue.cc:237] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba58a357c6b438b8630691660c7aa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35775 } }
I20260927 02:42:31.916604 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fba58a357c6b438b8630691660c7aa15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba58a357c6b438b8630691660c7aa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35775 } } }
I20260927 02:42:31.916699 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:31.917320 5589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:31.916956 5583 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fba58a357c6b438b8630691660c7aa15. Latest consensus state: current_term: 1 leader_uuid: "fba58a357c6b438b8630691660c7aa15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba58a357c6b438b8630691660c7aa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35775 } } }
I20260927 02:42:31.917728 5583 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:31.917973 5589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:31.919461 5589 catalog_manager.cc:1347] Generated new cluster ID: 447121580016427baca413d2806c260b
I20260927 02:42:31.919507 5589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:31.925388 5589 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:31.925990 5589 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:31.937405 5589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fba58a357c6b438b8630691660c7aa15: Generated new TSK 0
I20260927 02:42:31.937611 5589 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:42:31.968366 5606 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 02:42:31.968498 5606 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260927 02:42:31.969049 5605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:31.952603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35775 (local address 127.0.0.1:33302)
0927 02:42:31.952785 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.952794 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:31.952859 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:31.953058 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:31.953059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:31.953061 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:42:31.953118 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:31.953120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:31.954076 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:31.954077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:31.954717 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:31.954720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:31.955221 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:31.955225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:31.955278 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:31.955300 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:42:31.968592 (+ 13292us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:42:31.968715 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35775: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260927 02:42:31.969363 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.952686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.0.98.62:35775)
0927 02:42:31.952755 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:31.952757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:31.952851 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:42:31.952956 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:31.952957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:31.952970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:31.953055 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:42:31.953165 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:31.953973 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:31.954823 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:31.955152 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:31.955222 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:31.955417 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:42:31.968440 (+ 13023us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:42:31.969297 (+ 857us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 02:42:31.969427 5579 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35775: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 02:42:31.969820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 5509
2026-09-27T02:42:31Z chronyd exiting
I20260927 02:42:32.005255 392 test_util.cc:175] -----------------------------------------------
I20260927 02:42:32.005343 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476929252373-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 02:42:32.006242 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:42:32.006917 392 webserver.cc:460] Webserver started at https://127.0.0.1:39283/ using document root <none> and password file <none>
I20260927 02:42:32.006961 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:42:32.007143 392 webserver.cc:460] Webserver started at http://127.0.0.1:40535/ using document root <none> and password file <none>
2026-09-27T02:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:32Z Disabled control of system clock
I20260927 02:42:32.013906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43737
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43737
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39283/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/testchainca.cert with env {}
W20260927 02:42:32.093513 5612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:32.093760 5612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:32.093783 5612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:32.095192 5612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:32.095249 5612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:32.095261 5612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:32.095273 5612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:32.095283 5612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:42:32.095294 5612 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:42:32.095304 5612 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39283/jwks.json
I20260927 02:42:32.097007 5612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39283/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:32.097266 5612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:32.097478 5612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:32.102584 5618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:32.102557 5617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:32.102550 5620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:32.102886 5612 server_base.cc:1034] running on GCE node
I20260927 02:42:32.103137 5612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:32.103421 5612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:32.104560 5612 hybrid_clock.cc:648] HybridClock initialized: now 1790476952104545 us; error 34 us; skew 500 ppm
I20260927 02:42:32.105881 5612 webserver.cc:460] Webserver started at http://127.0.98.62:38645/ using document root <none> and password file <none>
I20260927 02:42:32.106184 5612 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:32.106235 5612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:32.106355 5612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:32.107420 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "a54ce793f1f546d4ae1fdd8439809d57"
format_stamp: "Formatted at 2026-09-27 02:42:32 on dist-test-slave-44kr"
server_key: "703d99b4ed6f393e8ba675fe5b1c0e9b"
server_key_iv: "bc2e9ba953985369c00c6073c0256db8"
server_key_version: "encryptionkey@0"
I20260927 02:42:32.107743 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "a54ce793f1f546d4ae1fdd8439809d57"
format_stamp: "Formatted at 2026-09-27 02:42:32 on dist-test-slave-44kr"
server_key: "703d99b4ed6f393e8ba675fe5b1c0e9b"
server_key_iv: "bc2e9ba953985369c00c6073c0256db8"
server_key_version: "encryptionkey@0"
I20260927 02:42:32.109144 5612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:32.109965 5626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:32.110107 5612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:32.110246 5612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "a54ce793f1f546d4ae1fdd8439809d57"
format_stamp: "Formatted at 2026-09-27 02:42:32 on dist-test-slave-44kr"
server_key: "703d99b4ed6f393e8ba675fe5b1c0e9b"
server_key_iv: "bc2e9ba953985369c00c6073c0256db8"
server_key_version: "encryptionkey@0"
I20260927 02:42:32.110312 5612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:32.126016 5612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:32.126331 5612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:32.126457 5612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:32.130264 5612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43737
I20260927 02:42:32.130352 5678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43737 every 8 connection(s)
I20260927 02:42:32.130650 5612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:32.131609 5679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:32.133983 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57: Bootstrap starting.
I20260927 02:42:32.134557 5679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:32.134833 5679 log.cc:824] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:32.135548 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57: No bootstrap required, opened a new log
I20260927 02:42:32.137171 5679 raft_consensus.cc:348] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54ce793f1f546d4ae1fdd8439809d57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } }
I20260927 02:42:32.137323 5679 raft_consensus.cc:374] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:32.137353 5679 raft_consensus.cc:729] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54ce793f1f546d4ae1fdd8439809d57, State: Initialized, Role: FOLLOWER
I20260927 02:42:32.137485 5679 consensus_queue.cc:260] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54ce793f1f546d4ae1fdd8439809d57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } }
I20260927 02:42:32.137552 5679 raft_consensus.cc:388] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:32.137583 5679 raft_consensus.cc:482] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:32.137630 5679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:32.138198 5679 raft_consensus.cc:504] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54ce793f1f546d4ae1fdd8439809d57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } }
I20260927 02:42:32.138216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 5612
I20260927 02:42:32.138298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:32.138302 5679 leader_election.cc:302] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a54ce793f1f546d4ae1fdd8439809d57; no voters:
I20260927 02:42:32.138455 5679 leader_election.cc:288] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:32.138473 392 external_mini_cluster.cc:1428] Setting key 5a17b39ec7451314a18c5fd4713624b1
I20260927 02:42:32.138499 5682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:32.138609 5682 raft_consensus.cc:686] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [term 1 LEADER]: Becoming Leader. State: Replica: a54ce793f1f546d4ae1fdd8439809d57, State: Running, Role: LEADER
I20260927 02:42:32.138732 5679 sys_catalog.cc:564] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:32.139081 5682 consensus_queue.cc:237] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54ce793f1f546d4ae1fdd8439809d57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } }
I20260927 02:42:32.139518 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a54ce793f1f546d4ae1fdd8439809d57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54ce793f1f546d4ae1fdd8439809d57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } } }
I20260927 02:42:32.139598 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:32.139652 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a54ce793f1f546d4ae1fdd8439809d57. Latest consensus state: current_term: 1 leader_uuid: "a54ce793f1f546d4ae1fdd8439809d57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54ce793f1f546d4ae1fdd8439809d57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } } }
I20260927 02:42:32.139714 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:32.139813 5693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:32.140321 5693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:32.141865 5693 catalog_manager.cc:1347] Generated new cluster ID: 57c34a92ae5941d88d32156df8661dd7
I20260927 02:42:32.141917 5693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:32.145565 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:32.139134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.98.62:43737)
0927 02:42:32.139372 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:32.139382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:32.140270 (+ 888us) server_negotiation.cc:407] Connection header received
0927 02:42:32.140626 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:32.140633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:32.140692 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:32.140785 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:32.141168 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.142691 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.143503 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.143976 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.144095 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:32.144781 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:32.144785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:32.144785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:32.145036 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:32.145072 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:32.145083 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:32.145139 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:32.145236 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 02:42:32.145327 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260927 02:42:32.146461 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:32.157449 5693 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:32.158036 5693 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:32.160898 5693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a54ce793f1f546d4ae1fdd8439809d57: Generated new TSK 0
I20260927 02:42:32.161098 5693 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:42:32.182837 5709 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 02:42:32.182915 5709 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:42:32.183285 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:32.167318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.0.98.62:43737)
0927 02:42:32.167381 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:32.167383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:32.167418 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:42:32.167491 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:32.167493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:32.167507 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:32.167780 (+ 273us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:42:32.167914 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.168770 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.169718 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.170067 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.170135 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:32.170407 (+ 272us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:42:32.183035 (+ 12628us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:42:32.183211 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:42:32.183347 5686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 02:42:32.183379 5708 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:32.167216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43737 (local address 127.0.0.1:35402)
0927 02:42:32.167350 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:32.167361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:32.167425 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:32.167780 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:32.167782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:32.167784 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:42:32.167842 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:32.167844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.168904 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.168905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:32.169628 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:32.169632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.170218 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.170224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:32.170291 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:32.170298 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:42:32.183178 (+ 12880us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:42:32.183290 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43737: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":89,"threads_started":1}
I20260927 02:42:32.183691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 5612
2026-09-27T02:42:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 02:42:32.220191 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:42:32.220958 392 webserver.cc:460] Webserver started at https://127.0.0.1:41457/ using document root <none> and password file <none>
I20260927 02:42:32.221004 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:42:32.221110 392 webserver.cc:460] Webserver started at http://127.0.0.1:46611/ using document root <none> and password file <none>
2026-09-27T02:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:32Z Disabled control of system clock
I20260927 02:42:32.228696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33577
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33577
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41457/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:42:32.307574 5715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:32.307801 5715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:32.307821 5715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:32.309089 5715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:32.309142 5715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:32.309154 5715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:32.309165 5715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:32.309175 5715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:42:32.309186 5715 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:42:32.309196 5715 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41457/jwks.json
I20260927 02:42:32.310684 5715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41457/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:32.310957 5715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:32.311261 5715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:32.315850 5721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:32.315903 5720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:32.315961 5723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:32.316293 5715 server_base.cc:1034] running on GCE node
I20260927 02:42:32.316562 5715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:32.316830 5715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:32.318001 5715 hybrid_clock.cc:648] HybridClock initialized: now 1790476952317985 us; error 30 us; skew 500 ppm
I20260927 02:42:32.319332 5715 webserver.cc:460] Webserver started at http://127.0.98.62:35455/ using document root <none> and password file <none>
I20260927 02:42:32.319557 5715 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:32.319602 5715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:32.319711 5715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:32.320797 5715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "4c697a1c05684778b01c3c9a16e43fb3"
format_stamp: "Formatted at 2026-09-27 02:42:32 on dist-test-slave-44kr"
server_key: "2fa6f82b59919bb075db10e82f205387"
server_key_iv: "272fc7e8a606ec772728b38279da9823"
server_key_version: "encryptionkey@0"
I20260927 02:42:32.321120 5715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c697a1c05684778b01c3c9a16e43fb3"
format_stamp: "Formatted at 2026-09-27 02:42:32 on dist-test-slave-44kr"
server_key: "2fa6f82b59919bb075db10e82f205387"
server_key_iv: "272fc7e8a606ec772728b38279da9823"
server_key_version: "encryptionkey@0"
I20260927 02:42:32.322372 5715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:32.323109 5729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:32.323305 5715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:32.323385 5715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "4c697a1c05684778b01c3c9a16e43fb3"
format_stamp: "Formatted at 2026-09-27 02:42:32 on dist-test-slave-44kr"
server_key: "2fa6f82b59919bb075db10e82f205387"
server_key_iv: "272fc7e8a606ec772728b38279da9823"
server_key_version: "encryptionkey@0"
I20260927 02:42:32.323438 5715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:32.336958 5715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:32.337260 5715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:32.337365 5715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:32.340942 5715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33577
I20260927 02:42:32.341033 5781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33577 every 8 connection(s)
I20260927 02:42:32.341331 5715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:32.342021 5782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:32.342943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 5715
I20260927 02:42:32.343031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:32.343206 392 external_mini_cluster.cc:1428] Setting key 058cd20173bbb19a5ff13ac2050a79ad
I20260927 02:42:32.345341 5782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3: Bootstrap starting.
I20260927 02:42:32.346009 5782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:32.346297 5782 log.cc:824] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:32.346947 5782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3: No bootstrap required, opened a new log
I20260927 02:42:32.349130 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:32.343688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.0.98.62:33577)
0927 02:42:32.343899 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:32.343908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:32.343942 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:42:32.344499 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:32.344509 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:32.344603 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:32.344762 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:32.344974 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.346236 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.347003 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.347596 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.347721 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:32.348362 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:32.348366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:32.348367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:32.348603 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:32.348644 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:32.348659 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:32.348751 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:32.348808 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 02:42:32.348900 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260927 02:42:32.349318 5782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c697a1c05684778b01c3c9a16e43fb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33577 } }
I20260927 02:42:32.349452 5782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:32.349471 5782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c697a1c05684778b01c3c9a16e43fb3, State: Initialized, Role: FOLLOWER
I20260927 02:42:32.349586 5782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c697a1c05684778b01c3c9a16e43fb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33577 } }
I20260927 02:42:32.349658 5782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:32.349694 5782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:32.349731 5782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:32.350190 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:32.350414 5782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c697a1c05684778b01c3c9a16e43fb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33577 } }
I20260927 02:42:32.350521 5782 leader_election.cc:302] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c697a1c05684778b01c3c9a16e43fb3; no voters:
I20260927 02:42:32.350697 5782 leader_election.cc:288] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:32.350750 5787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:32.350961 5782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:32.351018 5787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [term 1 LEADER]: Becoming Leader. State: Replica: 4c697a1c05684778b01c3c9a16e43fb3, State: Running, Role: LEADER
I20260927 02:42:32.351115 5787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c697a1c05684778b01c3c9a16e43fb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33577 } }
I20260927 02:42:32.351464 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c697a1c05684778b01c3c9a16e43fb3. Latest consensus state: current_term: 1 leader_uuid: "4c697a1c05684778b01c3c9a16e43fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c697a1c05684778b01c3c9a16e43fb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33577 } } }
I20260927 02:42:32.351538 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:32.351686 5788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c697a1c05684778b01c3c9a16e43fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c697a1c05684778b01c3c9a16e43fb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33577 } } }
I20260927 02:42:32.351753 5788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:32.351819 5795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:32.352365 5795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:32.353860 5795 catalog_manager.cc:1347] Generated new cluster ID: 6d6b8cec21b1411e9ed8702a211ae80e
I20260927 02:42:32.353915 5795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:32.359907 5795 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:32.360584 5795 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:32.367630 5795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c697a1c05684778b01c3c9a16e43fb3: Generated new TSK 0
I20260927 02:42:32.367815 5795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:32.378438 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:32.374594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.0.98.62:33577)
0927 02:42:32.374636 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:32.374638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:32.374754 (+ 116us) server_negotiation.cc:407] Connection header received
0927 02:42:32.374810 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:32.374811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:32.374824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:32.374906 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:42:32.374988 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.375789 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.376715 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.377036 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.377102 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:32.377235 (+ 133us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:42:32.378228 (+ 993us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:42:32.378364 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:42:32.378504 5785 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 02:42:32.380306 5811 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:32.374511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33577 (local address 127.0.0.1:59700)
0927 02:42:32.374684 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:32.374692 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:32.374750 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:32.374902 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:32.374902 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:32.374904 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:42:32.374949 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:32.374951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.375896 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.375896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:32.376642 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:32.376646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.377106 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.377109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:32.377148 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:32.377153 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:42:32.378328 (+ 1175us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:42:32.380195 (+ 1867us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33577: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1694,"thread_start_us":82,"threads_started":1}
I20260927 02:42:32.380517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 5715
E20260927 02:42:32.390918 5812 webserver.cc:533] Webserver: sslize failed:
E20260927 02:42:32.391004 5812 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T02:42:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (197 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T02:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:32Z Disabled control of system clock
I20260927 02:42:32.424752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39363
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39363
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/testchainca.cert with env {}
W20260927 02:42:32.503485 5816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:32.503688 5816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:32.503710 5816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:32.504954 5816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:32.505000 5816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:32.505012 5816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:32.505023 5816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:32.505033 5816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:42:32.505043 5816 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:42:32.505052 5816 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 02:42:32.506675 5816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:32.506923 5816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:32.507131 5816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:32.512036 5821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:32.512185 5816 server_base.cc:1034] running on GCE node
W20260927 02:42:32.512230 5822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:32.512030 5824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:32.512554 5816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:32.512853 5816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:32.514024 5816 hybrid_clock.cc:648] HybridClock initialized: now 1790476952513983 us; error 58 us; skew 500 ppm
I20260927 02:42:32.515290 5816 webserver.cc:460] Webserver started at http://127.0.98.62:34573/ using document root <none> and password file <none>
I20260927 02:42:32.515525 5816 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:32.515609 5816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:32.515744 5816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:32.516870 5816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "cb61d14848b547738bacdcb603074a60"
format_stamp: "Formatted at 2026-09-27 02:42:32 on dist-test-slave-44kr"
server_key: "2cec6077e8eeeb5fa112c3a7571b3be8"
server_key_iv: "30b621ad8fc1ab0080038f0f0f77fb8b"
server_key_version: "encryptionkey@0"
I20260927 02:42:32.517195 5816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb61d14848b547738bacdcb603074a60"
format_stamp: "Formatted at 2026-09-27 02:42:32 on dist-test-slave-44kr"
server_key: "2cec6077e8eeeb5fa112c3a7571b3be8"
server_key_iv: "30b621ad8fc1ab0080038f0f0f77fb8b"
server_key_version: "encryptionkey@0"
I20260927 02:42:32.518394 5816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:42:32.519101 5830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:32.519339 5816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:32.519428 5816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "cb61d14848b547738bacdcb603074a60"
format_stamp: "Formatted at 2026-09-27 02:42:32 on dist-test-slave-44kr"
server_key: "2cec6077e8eeeb5fa112c3a7571b3be8"
server_key_iv: "30b621ad8fc1ab0080038f0f0f77fb8b"
server_key_version: "encryptionkey@0"
I20260927 02:42:32.519495 5816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:32.538901 5816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:32.539206 5816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:32.539333 5816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:32.542929 5816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39363
I20260927 02:42:32.542973 5882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39363 every 8 connection(s)
I20260927 02:42:32.543279 5816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:32.544185 5883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:32.546473 5883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60: Bootstrap starting.
I20260927 02:42:32.547034 5883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:32.547289 5883 log.cc:824] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:32.547950 5883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60: No bootstrap required, opened a new log
I20260927 02:42:32.548691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 5816
I20260927 02:42:32.548781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:32.548941 392 external_mini_cluster.cc:1428] Setting key 06c64a5dc2c4c1758b38e98d7d3111c2
I20260927 02:42:32.549817 5883 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb61d14848b547738bacdcb603074a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39363 } }
I20260927 02:42:32.549958 5883 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:32.549989 5883 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb61d14848b547738bacdcb603074a60, State: Initialized, Role: FOLLOWER
I20260927 02:42:32.550122 5883 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb61d14848b547738bacdcb603074a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39363 } }
I20260927 02:42:32.550190 5883 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:32.550222 5883 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:32.550269 5883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:32.550801 5883 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb61d14848b547738bacdcb603074a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39363 } }
I20260927 02:42:32.550899 5883 leader_election.cc:302] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb61d14848b547738bacdcb603074a60; no voters:
I20260927 02:42:32.551054 5883 leader_election.cc:288] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:32.551090 5888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:32.551249 5888 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [term 1 LEADER]: Becoming Leader. State: Replica: cb61d14848b547738bacdcb603074a60, State: Running, Role: LEADER
I20260927 02:42:32.551309 5883 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:32.551330 5888 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb61d14848b547738bacdcb603074a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39363 } }
I20260927 02:42:32.551635 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb61d14848b547738bacdcb603074a60. Latest consensus state: current_term: 1 leader_uuid: "cb61d14848b547738bacdcb603074a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb61d14848b547738bacdcb603074a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39363 } } }
I20260927 02:42:32.551640 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb61d14848b547738bacdcb603074a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb61d14848b547738bacdcb603074a60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39363 } } }
I20260927 02:42:32.551707 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:32.551729 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:32.552997 5894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:32.553738 5894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:32.555019 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:32.549443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.0.98.62:39363)
0927 02:42:32.549651 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:32.549659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:32.549684 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:42:32.549780 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:32.549786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:32.549841 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:32.549958 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:32.550164 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.551678 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.552608 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.553299 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.553432 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:32.554256 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:32.554260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:32.554260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:32.554477 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:32.554519 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:32.554535 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:32.554635 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:32.554707 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:42:32.554801 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260927 02:42:32.555979 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:32.556892 5894 catalog_manager.cc:1347] Generated new cluster ID: a62acbb948f94b118502860691459297
I20260927 02:42:32.556944 5894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:32.565780 5894 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:32.566519 5894 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:32.569938 5894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb61d14848b547738bacdcb603074a60: Generated new TSK 0
I20260927 02:42:32.570139 5894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:32.716753 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:32.578642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.0.98.62:39363)
0927 02:42:32.578688 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:32.578690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:32.578752 (+ 62us) server_negotiation.cc:407] Connection header received
0927 02:42:32.578804 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:32.578805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:32.578817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:32.578902 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:42:32.578990 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.579763 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.580662 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.580969 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.581040 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:32.581303 (+ 263us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:42:32.716522 (+135219us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:42:32.716693 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32876: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 02:42:32.716817 5887 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32876: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 02:42:32.716918 5912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:32.578540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39363 (local address 127.0.0.1:32876)
0927 02:42:32.578682 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:32.578692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:32.578751 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:32.578900 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:32.578901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:32.578902 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:42:32.578954 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:32.578955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.579857 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.579858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:32.580586 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:32.580590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:32.581131 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:32.581134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:32.581195 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:32.581201 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:42:32.716671 (+135470us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:42:32.716801 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260927 02:42:32.717352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 5816
2026-09-27T02:42:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (316 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:32 dist-test-slave-44kr krb5kdc[5924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:32 dist-test-slave-44kr krb5kdc[5924](info): set up 2 sockets
Sep 27 02:42:32 dist-test-slave-44kr krb5kdc[5924](info): commencing operation
krb5kdc: starting...
W20260927 02:42:33.852481 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:33 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:33.976465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40831
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:34.055678 5937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:34.055907 5937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:34.055939 5937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:34.057204 5937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:34.057252 5937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:34.057271 5937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:34.057287 5937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:34.057302 5937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:34.058861 5937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:34.059109 5937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:34.059333 5937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:34.064154 5942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:34.064313 5945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:34.064349 5943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:34.064287 5937 server_base.cc:1034] running on GCE node
I20260927 02:42:34.064731 5937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:34.065032 5937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:34.066241 5937 hybrid_clock.cc:648] HybridClock initialized: now 1790476954066216 us; error 45 us; skew 500 ppm
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.069203 5937 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:34.069690 5937 webserver.cc:460] Webserver started at http://127.0.98.62:44765/ using document root <none> and password file <none>
I20260927 02:42:34.069924 5937 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:34.069976 5937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:34.070122 5937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:34.071290 5937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "b2387d8cc2f7413d9330ef3f0c832b4b"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "acb44faca15333f77b5109e9d4135dee"
server_key_iv: "ca1e50ac41ba576fdb0a307c10693d60"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.071650 5937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "b2387d8cc2f7413d9330ef3f0c832b4b"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "acb44faca15333f77b5109e9d4135dee"
server_key_iv: "ca1e50ac41ba576fdb0a307c10693d60"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.073117 5937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:34.074167 5952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.074476 5937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:34.074576 5937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "b2387d8cc2f7413d9330ef3f0c832b4b"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "acb44faca15333f77b5109e9d4135dee"
server_key_iv: "ca1e50ac41ba576fdb0a307c10693d60"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.074648 5937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:34.091912 5937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:34.094228 5937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:34.094374 5937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:34.098275 5937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40831
I20260927 02:42:34.098326 6004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40831 every 8 connection(s)
I20260927 02:42:34.098819 5937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:34.099474 6005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:34.101282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 5937
I20260927 02:42:34.101387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:34.101548 392 external_mini_cluster.cc:1428] Setting key 869e65868b7919dd517b23c3fe3977c4
I20260927 02:42:34.101948 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b: Bootstrap starting.
I20260927 02:42:34.102595 6005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:34.102875 6005 log.cc:824] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:34.103498 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b: No bootstrap required, opened a new log
I20260927 02:42:34.105361 6005 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2387d8cc2f7413d9330ef3f0c832b4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40831 } }
I20260927 02:42:34.105500 6005 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:34.105532 6005 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2387d8cc2f7413d9330ef3f0c832b4b, State: Initialized, Role: FOLLOWER
I20260927 02:42:34.105650 6005 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2387d8cc2f7413d9330ef3f0c832b4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40831 } }
I20260927 02:42:34.105721 6005 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:34.105751 6005 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:34.105799 6005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:34.106372 6005 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2387d8cc2f7413d9330ef3f0c832b4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40831 } }
I20260927 02:42:34.106467 6005 leader_election.cc:302] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2387d8cc2f7413d9330ef3f0c832b4b; no voters:
I20260927 02:42:34.106633 6005 leader_election.cc:288] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:34.106670 6010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:34.106777 6010 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [term 1 LEADER]: Becoming Leader. State: Replica: b2387d8cc2f7413d9330ef3f0c832b4b, State: Running, Role: LEADER
I20260927 02:42:34.106868 6010 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2387d8cc2f7413d9330ef3f0c832b4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40831 } }
I20260927 02:42:34.106900 6005 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:34.107213 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2387d8cc2f7413d9330ef3f0c832b4b. Latest consensus state: current_term: 1 leader_uuid: "b2387d8cc2f7413d9330ef3f0c832b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2387d8cc2f7413d9330ef3f0c832b4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40831 } } }
I20260927 02:42:34.107291 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:34.107226 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2387d8cc2f7413d9330ef3f0c832b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2387d8cc2f7413d9330ef3f0c832b4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40831 } } }
I20260927 02:42:34.107450 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b [sys.catalog]: This master's current role is: LEADER
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260927 02:42:34.108609 6025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:34.108706 6025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:34.109036 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:34.109565 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:34.111375 6015 catalog_manager.cc:1347] Generated new cluster ID: 66619292d0584f38b2e061212b0b0015
I20260927 02:42:34.111429 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:34.112164 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.102123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.0.98.62:40831)
0927 02:42:34.102305 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.102313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.102339 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:42:34.102796 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.102804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.102855 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.103002 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.103622 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.104596 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.105331 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.105787 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.105887 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.108622 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:34.108626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:34.108627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:34.108639 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:34.110405 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.110853 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.110857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.110857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.110985 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.111199 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.111200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.111201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.111523 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:34.111715 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:34.111817 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:42:34.111953 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260927 02:42:34.116374 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:34.117002 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:34.121131 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2387d8cc2f7413d9330ef3f0c832b4b: Generated new TSK 0
I20260927 02:42:34.121320 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:34.151770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40831
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:34.247972 6031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:34.248198 6031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:34.248227 6031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:34.249454 6031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:34.249512 6031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:34.249542 6031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:34.251106 6031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:34.251374 6031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:34.251650 6031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:34.256695 6037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:34.256814 6036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:34.256877 6031 server_base.cc:1034] running on GCE node
W20260927 02:42:34.256960 6039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:34.257239 6031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:34.257498 6031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:34.258680 6031 hybrid_clock.cc:648] HybridClock initialized: now 1790476954258667 us; error 27 us; skew 500 ppm
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.261514 6031 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:34.262048 6031 webserver.cc:460] Webserver started at http://127.0.98.1:35389/ using document root <none> and password file <none>
I20260927 02:42:34.262269 6031 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:34.262319 6031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:34.262447 6031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:34.263518 6031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "b4e5c2dcb4b4439e98f94df692a2e771"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "4caf8db1ee7d9afd389d01061982e1c0"
server_key_iv: "c304b908612bfa3d1c82a6cc8edb8a7f"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.263846 6031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b4e5c2dcb4b4439e98f94df692a2e771"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "4caf8db1ee7d9afd389d01061982e1c0"
server_key_iv: "c304b908612bfa3d1c82a6cc8edb8a7f"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.265252 6031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:34.266080 6046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.266290 6031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:34.266378 6031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "b4e5c2dcb4b4439e98f94df692a2e771"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "4caf8db1ee7d9afd389d01061982e1c0"
server_key_iv: "c304b908612bfa3d1c82a6cc8edb8a7f"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.266443 6031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:34.276774 6031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:34.279348 6031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:34.279481 6031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:34.279711 6031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:34.280030 6031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:34.280092 6031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.280122 6031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:34.280138 6031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.286090 6031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36125
I20260927 02:42:34.286182 6159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36125 every 8 connection(s)
I20260927 02:42:34.286477 6031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:34.287447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 6031
I20260927 02:42:34.287557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:34.287739 392 external_mini_cluster.cc:1428] Setting key 6685a79bc457b0d712b72b2c33a8cbea
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:42:34.297194 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.287305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37071 (local address 127.0.98.62:40831)
0927 02:42:34.287358 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.287360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.287549 (+ 189us) server_negotiation.cc:407] Connection header received
0927 02:42:34.288791 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.288792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.288801 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.288840 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.289682 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.290253 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.291322 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.291525 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.291583 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.293979 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:34.293981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:34.293981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:34.293988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:34.295638 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.296074 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.296075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.296076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.296178 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.296483 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.296483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.296484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.296607 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:34.296663 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:34.296989 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 02:42:34.297074 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:42:34.297277 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.287182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40831 (local address 127.0.98.1:37071)
0927 02:42:34.287380 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:34.287433 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:34.288684 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:34.288932 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:34.288937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:34.289288 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:34.289579 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:34.289586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.290358 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.290359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:34.291233 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:34.291235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.291603 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.291604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:34.291734 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.292314 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:34.292323 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:34.293855 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:34.295742 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:34.295749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:34.295752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:34.295999 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:34.296265 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:34.296266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:34.296267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:34.296426 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:34.296680 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:34.296688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:34.296822 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 02:42:34.296910 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 02:42:34.297047 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260927 02:42:34.297844 6160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40831
I20260927 02:42:34.297974 6160 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:42:34.298166 6160 heartbeater.cc:511] Master 127.0.98.62:40831 requested a full tablet report, sending...
I20260927 02:42:34.298748 5969 ts_manager.cc:194] Registered new tserver with Master: b4e5c2dcb4b4439e98f94df692a2e771 (127.0.98.1:36125)
I20260927 02:42:34.299603 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:34.315815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40831
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:34.396616 6165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:34.396835 6165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:34.396858 6165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:34.398043 6165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:34.398087 6165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:34.398111 6165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:34.399621 6165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:34.399823 6165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:34.400000 6165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:34.405126 6173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:34.405102 6170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:34.405164 6165 server_base.cc:1034] running on GCE node
W20260927 02:42:34.405102 6171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:34.405506 6165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:34.405791 6165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:34.406970 6165 hybrid_clock.cc:648] HybridClock initialized: now 1790476954406951 us; error 28 us; skew 500 ppm
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.410133 6165 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:34.410652 6165 webserver.cc:460] Webserver started at http://127.0.98.2:33465/ using document root <none> and password file <none>
I20260927 02:42:34.410852 6165 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:34.410902 6165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:34.411026 6165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:34.412084 6165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "a7c63e800325430eb82772b4e2b8398b"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "4f21d8e92e611500996ed22d3cb9ae0f"
server_key_iv: "b5882a5aa6cadb88ead8373c5a48f1ff"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.412480 6165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a7c63e800325430eb82772b4e2b8398b"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "4f21d8e92e611500996ed22d3cb9ae0f"
server_key_iv: "b5882a5aa6cadb88ead8373c5a48f1ff"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.413748 6165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:34.414580 6180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.414778 6165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:34.414861 6165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "a7c63e800325430eb82772b4e2b8398b"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "4f21d8e92e611500996ed22d3cb9ae0f"
server_key_iv: "b5882a5aa6cadb88ead8373c5a48f1ff"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.414927 6165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:34.434316 6165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:34.437021 6165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:34.437230 6165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:34.437474 6165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:34.437776 6165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:34.437808 6165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.437839 6165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:34.437855 6165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.443861 6165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42287
I20260927 02:42:34.443908 6293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42287 every 8 connection(s)
I20260927 02:42:34.444231 6165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:34.451081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 6165
I20260927 02:42:34.451188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:34.451370 392 external_mini_cluster.cc:1428] Setting key 650bf2c3044b3f2ab344f80716938425
I20260927 02:42:34.453970 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.445012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42643 (local address 127.0.98.62:40831)
0927 02:42:34.445088 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.445090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.445287 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:42:34.446245 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.446246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.446259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.446325 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.446968 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.447477 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.448517 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.448725 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.448789 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.450973 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:34.450974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:34.450974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:34.450981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:34.452470 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.452910 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.452911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.452911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.452960 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.453248 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.453249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.453250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.453367 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:34.453421 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:34.453789 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 02:42:34.453876 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:42:34.454043 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.444977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40831 (local address 127.0.98.2:42643)
0927 02:42:34.445157 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:34.445195 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:34.446123 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:34.446399 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:34.446403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:34.446667 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:34.446865 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:34.446869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.447580 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.447580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:34.448425 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:34.448427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.448803 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.448804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:34.448926 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.449407 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:34.449416 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:34.450868 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:34.452542 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:34.452548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:34.452551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:34.452839 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:34.453024 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:34.453025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:34.453025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:34.453159 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:34.453428 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:34.453432 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:34.453607 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 02:42:34.453747 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 02:42:34.453872 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260927 02:42:34.454514 6294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40831
I20260927 02:42:34.454605 6294 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:34.454793 6294 heartbeater.cc:511] Master 127.0.98.62:40831 requested a full tablet report, sending...
I20260927 02:42:34.455196 5969 ts_manager.cc:194] Registered new tserver with Master: a7c63e800325430eb82772b4e2b8398b (127.0.98.2:42287)
I20260927 02:42:34.455551 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42643
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:34.479660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40831
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:34.560369 6299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:34.560616 6299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:34.560642 6299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:34.561839 6299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:34.561882 6299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:34.561908 6299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:34.563416 6299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:34.563625 6299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:34.563800 6299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:34.568884 6305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:34.568888 6304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:34.568888 6307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:34.568904 6299 server_base.cc:1034] running on GCE node
I20260927 02:42:34.569339 6299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:34.569612 6299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:34.570770 6299 hybrid_clock.cc:648] HybridClock initialized: now 1790476954570754 us; error 56 us; skew 500 ppm
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.573361 6299 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:34.573818 6299 webserver.cc:460] Webserver started at http://127.0.98.3:39625/ using document root <none> and password file <none>
I20260927 02:42:34.574034 6299 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:34.574079 6299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:34.574203 6299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:34.575323 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "475ce53196cf4b11b06f345e59e53d0a"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "2791b27996aafb4ae7963c061c6f7f60"
server_key_iv: "f766f8a7b9734f8bfbd9dcc9c6ab3a37"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.575651 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "475ce53196cf4b11b06f345e59e53d0a"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "2791b27996aafb4ae7963c061c6f7f60"
server_key_iv: "f766f8a7b9734f8bfbd9dcc9c6ab3a37"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.577000 6299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:34.578009 6314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.578267 6299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:34.578353 6299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "475ce53196cf4b11b06f345e59e53d0a"
format_stamp: "Formatted at 2026-09-27 02:42:34 on dist-test-slave-44kr"
server_key: "2791b27996aafb4ae7963c061c6f7f60"
server_key_iv: "f766f8a7b9734f8bfbd9dcc9c6ab3a37"
server_key_version: "encryptionkey@0"
I20260927 02:42:34.578428 6299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:34.594834 6299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:34.597286 6299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:34.597405 6299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:34.597607 6299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:34.597895 6299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:34.597924 6299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.597944 6299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:34.597955 6299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:34.603724 6299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37613
I20260927 02:42:34.603778 6427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37613 every 8 connection(s)
I20260927 02:42:34.604095 6299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:34.604895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 6299
I20260927 02:42:34.605005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:34.605180 392 external_mini_cluster.cc:1428] Setting key 0dbb9853bc80d160cdbc162c3645554a
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:34.614377 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.604803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52259 (local address 127.0.98.62:40831)
0927 02:42:34.604846 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.604848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.605093 (+ 245us) server_negotiation.cc:407] Connection header received
0927 02:42:34.606346 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.606347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.606358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.606421 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.607178 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.607754 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.608971 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.609160 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.609225 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.611511 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:34.611512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:34.611512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:34.611519 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:34.612903 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.613387 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.613388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.613389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.613448 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.613726 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.613728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.613728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.613862 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:34.613933 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:34.614235 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 02:42:34.614309 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:42:34.614499 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.604781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40831 (local address 127.0.98.3:52259)
0927 02:42:34.604958 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:34.605009 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:34.606226 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:34.606484 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:34.606487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:34.606824 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:34.607073 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:34.607078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.607833 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.607834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:34.608881 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:34.608883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.609256 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.609257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:34.609372 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.609883 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:34.609892 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:34.611417 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:34.612993 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:34.613000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:34.613002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:34.613299 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:34.613524 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:34.613525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:34.613525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:34.613638 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:34.613926 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:34.613929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:34.614048 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 02:42:34.614179 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 02:42:34.614311 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260927 02:42:34.615242 6428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40831
I20260927 02:42:34.615345 6428 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:34.615517 6428 heartbeater.cc:511] Master 127.0.98.62:40831 requested a full tablet report, sending...
I20260927 02:42:34.615942 5969 ts_manager.cc:194] Registered new tserver with Master: 475ce53196cf4b11b06f345e59e53d0a (127.0.98.3:37613)
I20260927 02:42:34.616441 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52259
I20260927 02:42:34.617362 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/test-admin.keytab.
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.634572 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 02:42:34.636415 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:34.635598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.0.98.1:36125)
0927 02:42:34.635772 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.635776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.635786 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:34.635813 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.635814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.635828 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.635901 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.636299 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41366: BlockingRecv error: recv got EOF from 127.0.0.1:41366 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
W20260927 02:42:34.636543 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.637656 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.638684 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:34.647490 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.639328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.0.98.1:36125)
0927 02:42:34.639379 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.639380 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.639389 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:34.639453 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.639454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.639464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.639519 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.639980 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.640844 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.641665 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.642004 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.642077 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.644585 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:34.644594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:34.644594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:34.644602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:34.646184 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.646584 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.646587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.646587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.646651 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.646851 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.646852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.646852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.647015 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:34.647198 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:34.647335 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 02:42:34.647407 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 02:42:34.649505 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:34.648899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.0.98.1:36125)
0927 02:42:34.648951 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.648953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.648978 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:42:34.649048 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.649048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.649058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.649103 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.649457 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41372: BlockingRecv error: recv got EOF from 127.0.0.1:41372 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 02:42:34.649619 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.650499 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.651278 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:34.658036 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.651780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.0.98.1:36125)
0927 02:42:34.651827 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.651829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.651837 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:34.651877 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.651878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.651888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.651931 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.652262 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.652835 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.653549 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.653757 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.653809 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.655757 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:34.655758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:34.655758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:34.655765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:34.656995 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.657353 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.657354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.657355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.657404 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.657566 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.657567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.657567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.657668 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:34.657751 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:34.657865 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 02:42:34.657942 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 02:42:34.659608 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:34.659012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.0.98.1:36125)
0927 02:42:34.659043 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.659045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.659135 (+ 90us) server_negotiation.cc:407] Connection header received
0927 02:42:34.659179 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.659179 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.659186 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.659230 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.659559 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41388: BlockingRecv error: recv got EOF from 127.0.0.1:41388 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 02:42:34.659749 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476929252373-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.660748 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:34.661527 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:42:34.667902 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.662013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.0.98.1:36125)
0927 02:42:34.662063 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:34.662065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:34.662073 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:34.662132 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:34.662132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:34.662141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:34.662190 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:34.662559 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.663036 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.663717 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:34.663896 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:34.663949 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:34.665812 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:34.665813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:34.665814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:34.665821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:34.666942 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.667238 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.667239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.667239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.667288 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:34.667447 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:34.667447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:34.667448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:34.667540 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:34.667620 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:34.667738 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 02:42:34.667805 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:42:34.668521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 6031
I20260927 02:42:34.673377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 6165
I20260927 02:42:34.677897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 6299
I20260927 02:42:34.681672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 5937
2026-09-27T02:42:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1790 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[6448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[6448](info): set up 2 sockets
Sep 27 02:42:34 dist-test-slave-44kr krb5kdc[6448](info): commencing operation
krb5kdc: starting...
W20260927 02:42:35.648509 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:35 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:42:35.767581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43607
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:35.848219 6461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:35.848448 6461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:35.848470 6461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:35.849712 6461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:35.849762 6461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:35.849776 6461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:35.849786 6461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:35.849797 6461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:35.851266 6461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:35.851501 6461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:35.851740 6461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:35.856662 6469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:35.856647 6467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:35.856729 6461 server_base.cc:1034] running on GCE node
W20260927 02:42:35.856650 6466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:35.857290 6461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:35.857605 6461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:35.858775 6461 hybrid_clock.cc:648] HybridClock initialized: now 1790476955858759 us; error 33 us; skew 500 ppm
Sep 27 02:42:35 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476955, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:35.865111 6461 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:42:35.865648 6461 webserver.cc:460] Webserver started at http://127.0.98.62:43329/ using document root <none> and password file <none>
I20260927 02:42:35.865875 6461 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:35.865923 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:35.866065 6461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:35.867130 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "6592263c8e1d49dd9be3ddc85521a8c0"
format_stamp: "Formatted at 2026-09-27 02:42:35 on dist-test-slave-44kr"
server_key: "5dcd3a277c055a4ad94d106cd10af0ee"
server_key_iv: "e179fabe0f478455d54e808ed3cf2d7f"
server_key_version: "encryptionkey@0"
I20260927 02:42:35.867478 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "6592263c8e1d49dd9be3ddc85521a8c0"
format_stamp: "Formatted at 2026-09-27 02:42:35 on dist-test-slave-44kr"
server_key: "5dcd3a277c055a4ad94d106cd10af0ee"
server_key_iv: "e179fabe0f478455d54e808ed3cf2d7f"
server_key_version: "encryptionkey@0"
I20260927 02:42:35.868844 6461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:35.869668 6476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:35.869998 6461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:35.870121 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "6592263c8e1d49dd9be3ddc85521a8c0"
format_stamp: "Formatted at 2026-09-27 02:42:35 on dist-test-slave-44kr"
server_key: "5dcd3a277c055a4ad94d106cd10af0ee"
server_key_iv: "e179fabe0f478455d54e808ed3cf2d7f"
server_key_version: "encryptionkey@0"
I20260927 02:42:35.870203 6461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:35.900138 6461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:35.902631 6461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:35.902799 6461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:35.906780 6461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43607
I20260927 02:42:35.906876 6528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43607 every 8 connection(s)
I20260927 02:42:35.907186 6461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:35.908157 6529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:35.910559 6529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: Bootstrap starting.
I20260927 02:42:35.911124 6529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:35.911366 6529 log.cc:824] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:35.912075 6529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: No bootstrap required, opened a new log
I20260927 02:42:35.912937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 6461
I20260927 02:42:35.913050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:35.913223 392 external_mini_cluster.cc:1428] Setting key 77e7100d562f7060f3673a46fb20dac4
I20260927 02:42:35.914327 6529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } }
I20260927 02:42:35.914515 6529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:35.914554 6529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6592263c8e1d49dd9be3ddc85521a8c0, State: Initialized, Role: FOLLOWER
I20260927 02:42:35.914700 6529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } }
I20260927 02:42:35.914817 6529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:35.914860 6529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:35.914915 6529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:35.915928 6529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } }
I20260927 02:42:35.916025 6529 leader_election.cc:302] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6592263c8e1d49dd9be3ddc85521a8c0; no voters:
I20260927 02:42:35.916188 6529 leader_election.cc:288] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:35.916239 6534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:35.916414 6534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6592263c8e1d49dd9be3ddc85521a8c0, State: Running, Role: LEADER
I20260927 02:42:35.916500 6529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:35.916522 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } }
I20260927 02:42:35.916911 6536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6592263c8e1d49dd9be3ddc85521a8c0. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } }
I20260927 02:42:35.916987 6536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:35.917119 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } }
I20260927 02:42:35.917166 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:35.917282 6542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:35.917791 6542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:35.919275 6542 catalog_manager.cc:1347] Generated new cluster ID: e2e9a06fcbd5484fa0b2e57309ce8ba0
I20260927 02:42:35.919322 6542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:42:35 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:35.924723 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:35.913801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.0.98.62:43607)
0927 02:42:35.914000 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:35.914009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:35.914039 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:42:35.914270 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:35.914276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:35.914325 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:35.914447 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:35.915270 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:35.916254 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:35.917326 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:35.917998 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:35.918109 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:35.920834 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:35.920838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:35.920838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:35.920850 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:35.922727 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:35.923076 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:35.923079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:35.923079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:35.923262 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:35.923429 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:35.923430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:35.923430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:35.923723 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:35.923848 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:35.924337 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 02:42:35.924516 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260927 02:42:35.937036 6542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:35.937613 6542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:35.940572 6542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: Generated new TSK 0
I20260927 02:42:35.940747 6542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:42:35.973727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43607
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:36.055192 6555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:36.055392 6555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:36.055413 6555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:36.056529 6555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:36.056564 6555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:36.056589 6555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:36.057967 6555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:36.058157 6555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:36.058326 6555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:36.063153 6563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.063117 6561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.063230 6560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:36.063340 6555 server_base.cc:1034] running on GCE node
I20260927 02:42:36.063514 6555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:36.063752 6555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:36.064922 6555 hybrid_clock.cc:648] HybridClock initialized: now 1790476956064910 us; error 32 us; skew 500 ppm
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:36.067546 6555 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:42:36.068015 6555 webserver.cc:460] Webserver started at http://127.0.98.1:36291/ using document root <none> and password file <none>
I20260927 02:42:36.068223 6555 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:36.068269 6555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:36.068372 6555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:36.069456 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "5db7bf4890bd47d38125dd42e242de88"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "b7711e9827be10d553c6f874a6027fe4"
server_key_iv: "30c68f868ee3c2cf1134d1bd437f5753"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.069777 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5db7bf4890bd47d38125dd42e242de88"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "b7711e9827be10d553c6f874a6027fe4"
server_key_iv: "30c68f868ee3c2cf1134d1bd437f5753"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.071035 6555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:42:36.071851 6570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.072009 6555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:42:36.072093 6555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "5db7bf4890bd47d38125dd42e242de88"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "b7711e9827be10d553c6f874a6027fe4"
server_key_iv: "30c68f868ee3c2cf1134d1bd437f5753"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.072161 6555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:36.079635 6555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:36.081743 6555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:36.081875 6555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:36.082093 6555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:36.082413 6555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:36.082448 6555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.082476 6555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:36.082494 6555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.088074 6555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44895
I20260927 02:42:36.088135 6683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44895 every 8 connection(s)
I20260927 02:42:36.088482 6555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:36.098143 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.089342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54425 (local address 127.0.98.62:43607)
0927 02:42:36.089419 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.089422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.089662 (+ 240us) server_negotiation.cc:407] Connection header received
0927 02:42:36.090196 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.090197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.090210 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.090256 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.091215 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.091718 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.092921 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.093134 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.093198 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.095471 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.095472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.095473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.095480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.096876 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.097253 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.097254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.097255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.097303 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.097522 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.097523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.097523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.097632 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.097687 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.097942 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.098035 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:42:36.098304 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.089283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.1:54425)
0927 02:42:36.089485 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.089566 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.090107 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.090328 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.090331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.090916 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.091115 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.091119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.091819 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.091820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.092819 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.092822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.093236 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.093238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.093347 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.093830 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.093839 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.095363 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.096957 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.096964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.096966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.097181 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.097359 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.097360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.097360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.097463 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.097690 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.097696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.097826 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.097912 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.098065 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260927 02:42:36.098397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 6555
I20260927 02:42:36.098501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:36.098718 392 external_mini_cluster.cc:1428] Setting key 9d5b34b20d943aff79ecd25e8c2855ce
I20260927 02:42:36.098789 6684 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43607
I20260927 02:42:36.098887 6684 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:36.099080 6684 heartbeater.cc:511] Master 127.0.98.62:43607 requested a full tablet report, sending...
I20260927 02:42:36.099593 6493 ts_manager.cc:194] Registered new tserver with Master: 5db7bf4890bd47d38125dd42e242de88 (127.0.98.1:44895)
I20260927 02:42:36.100549 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54425
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:42:36.123950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43607
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:36.203639 6689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:36.203868 6689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:36.203893 6689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:36.205109 6689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:36.205153 6689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:36.205180 6689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:36.206650 6689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:36.206874 6689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:36.207095 6689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:36.211860 6694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.211911 6695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.211930 6697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:36.212030 6689 server_base.cc:1034] running on GCE node
I20260927 02:42:36.212323 6689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:36.212592 6689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:36.213761 6689 hybrid_clock.cc:648] HybridClock initialized: now 1790476956213737 us; error 40 us; skew 500 ppm
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:36.216734 6689 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:42:36.217262 6689 webserver.cc:460] Webserver started at http://127.0.98.2:35347/ using document root <none> and password file <none>
I20260927 02:42:36.217478 6689 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:36.217535 6689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:36.217665 6689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:36.218755 6689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "d363b47e300a45abaab6399f1182ea65"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "5bb4425c6fee31e396f80d1a781c0c30"
server_key_iv: "8acb7949179f4ac09a863ad6e860ad7a"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.219087 6689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d363b47e300a45abaab6399f1182ea65"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "5bb4425c6fee31e396f80d1a781c0c30"
server_key_iv: "8acb7949179f4ac09a863ad6e860ad7a"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.220425 6689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:42:36.221293 6704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.221491 6689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:36.221580 6689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "d363b47e300a45abaab6399f1182ea65"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "5bb4425c6fee31e396f80d1a781c0c30"
server_key_iv: "8acb7949179f4ac09a863ad6e860ad7a"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.221644 6689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:36.237700 6689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:36.239957 6689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:36.240137 6689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:36.240338 6689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:36.240657 6689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:36.240693 6689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.240715 6689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:36.240731 6689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.246241 6689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40497
I20260927 02:42:36.246294 6817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40497 every 8 connection(s)
I20260927 02:42:36.246606 6689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:36.248857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 6689
I20260927 02:42:36.248965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:36.249136 392 external_mini_cluster.cc:1428] Setting key 719e687645c41bc9bcd227305236261a
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:36.256299 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.247280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48575 (local address 127.0.98.62:43607)
0927 02:42:36.247324 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.247326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.247541 (+ 215us) server_negotiation.cc:407] Connection header received
0927 02:42:36.248054 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.248056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.248069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.248197 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.249308 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.249910 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.250922 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.251209 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.251275 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.253616 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.253618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.253618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.253626 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.254981 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.255376 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.255378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.255378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.255429 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.255665 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.255666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.255666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.255778 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.255846 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.256125 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.256209 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:42:36.256496 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.247249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:48575)
0927 02:42:36.247412 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.247450 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.247952 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.248294 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.248299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.248971 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.249221 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.249228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.249999 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.250001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.250833 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.250836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.251300 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.251301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.251425 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.251946 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.251954 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.253522 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.255074 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.255082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.255084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.255301 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.255489 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.255490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.255490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.255604 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.255850 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.255856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.255994 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.256085 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.256243 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:42:36.256965 6818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43607
I20260927 02:42:36.257055 6818 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:36.257230 6818 heartbeater.cc:511] Master 127.0.98.62:43607 requested a full tablet report, sending...
I20260927 02:42:36.257578 6493 ts_manager.cc:194] Registered new tserver with Master: d363b47e300a45abaab6399f1182ea65 (127.0.98.2:40497)
I20260927 02:42:36.257984 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:42:36.276697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43607
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:36.357200 6823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:36.357425 6823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:36.357455 6823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:36.358595 6823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:36.358636 6823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:36.358671 6823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:36.360083 6823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:36.360327 6823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:36.360569 6823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:36.365403 6829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.365408 6831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.365420 6828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:36.365590 6823 server_base.cc:1034] running on GCE node
I20260927 02:42:36.365779 6823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:36.365972 6823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:36.367115 6823 hybrid_clock.cc:648] HybridClock initialized: now 1790476956367097 us; error 28 us; skew 500 ppm
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:36.369539 6823 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:42:36.370014 6823 webserver.cc:460] Webserver started at http://127.0.98.3:36843/ using document root <none> and password file <none>
I20260927 02:42:36.370226 6823 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:36.370272 6823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:36.370388 6823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:36.371385 6823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "bd77f6002a0941d7acd2243f0b8f78fa"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "556e2c0dd5d6c0f80e059036cd0bb7a6"
server_key_iv: "849b52431614c6d9c168793ada1dcd0f"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.371702 6823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bd77f6002a0941d7acd2243f0b8f78fa"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "556e2c0dd5d6c0f80e059036cd0bb7a6"
server_key_iv: "849b52431614c6d9c168793ada1dcd0f"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.372956 6823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:36.373718 6838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.373891 6823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.373968 6823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "bd77f6002a0941d7acd2243f0b8f78fa"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "556e2c0dd5d6c0f80e059036cd0bb7a6"
server_key_iv: "849b52431614c6d9c168793ada1dcd0f"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.374034 6823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:36.383852 6823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:36.386092 6823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:36.386212 6823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:36.386448 6823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:36.386745 6823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:36.386780 6823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.386801 6823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:36.386812 6823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.392474 6823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43277
I20260927 02:42:36.392514 6951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43277 every 8 connection(s)
I20260927 02:42:36.392850 6823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:36.401914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 6823
I20260927 02:42:36.402033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:36.402211 392 external_mini_cluster.cc:1428] Setting key 7f440627fffcead2242fba1ce7219d8c
I20260927 02:42:36.402592 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.393680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46277 (local address 127.0.98.62:43607)
0927 02:42:36.393739 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.393742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.393896 (+ 154us) server_negotiation.cc:407] Connection header received
0927 02:42:36.394400 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.394401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.394413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.394474 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.395488 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.396026 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.397118 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.397311 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.397374 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.399513 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.399514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.399515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.399521 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.400911 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.401375 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.401376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.401376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.401424 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.401723 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.401724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.401725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.401904 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.401982 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.402372 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.402458 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:42:36.402750 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.393600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.3:46277)
0927 02:42:36.393762 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.393807 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.394281 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.394533 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.394537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.395133 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.395367 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.395373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.396121 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.396122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.397028 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.397031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.397390 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.397390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.397496 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.397968 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.397977 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.399424 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.400998 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.401004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.401006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.401308 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.401500 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.401501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.401501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.401625 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.402004 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.402022 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.402225 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.402347 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.402520 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260927 02:42:36.403221 6952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43607
I20260927 02:42:36.403328 6952 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:36.403497 6952 heartbeater.cc:511] Master 127.0.98.62:43607 requested a full tablet report, sending...
I20260927 02:42:36.403926 6493 ts_manager.cc:194] Registered new tserver with Master: bd77f6002a0941d7acd2243f0b8f78fa (127.0.98.3:43277)
I20260927 02:42:36.404304 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46277
I20260927 02:42:36.414112 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 02:42:36.440212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33363
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43607,127.0.98.61:33363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:36.520970 6957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:36.521203 6957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:36.521235 6957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:36.522420 6957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:36.522466 6957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:36.522485 6957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:36.522501 6957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:36.522516 6957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:36.524053 6957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43607,127.0.98.61:33363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
W20260927 02:42:36.524174 6957 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43607,127.0.98.61:33363'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 02:42:36.524344 6957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:36.524605 6957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:36.529323 6963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.529337 6965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:36.529496 6957 server_base.cc:1034] running on GCE node
W20260927 02:42:36.529523 6962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:36.529807 6957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:36.530125 6957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:36.531293 6957 hybrid_clock.cc:648] HybridClock initialized: now 1790476956531274 us; error 40 us; skew 500 ppm
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:36.533828 6957 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:42:36.534394 6957 webserver.cc:460] Webserver started at http://127.0.98.61:37955/ using document root <none> and password file <none>
I20260927 02:42:36.534615 6957 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:36.534668 6957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:36.534822 6957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:36.535871 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data/instance:
uuid: "9c329716d8dd4447b1c8b9206a987619"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "b163405a7c8458bb5bc248521511fb0c"
server_key_iv: "0b9138b81006687111845995bdc6e5ca"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.536196 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal/instance:
uuid: "9c329716d8dd4447b1c8b9206a987619"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "b163405a7c8458bb5bc248521511fb0c"
server_key_iv: "0b9138b81006687111845995bdc6e5ca"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.537513 6957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:36.538339 6972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.538528 6957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:36.538648 6957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal
uuid: "9c329716d8dd4447b1c8b9206a987619"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "b163405a7c8458bb5bc248521511fb0c"
server_key_iv: "0b9138b81006687111845995bdc6e5ca"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.538721 6957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:36.551824 6957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:36.554088 6957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:36.554245 6957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:36.558218 6957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33363
I20260927 02:42:36.558301 7024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33363 every 8 connection(s)
I20260927 02:42:36.558609 6957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:42:36.559572 7025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:36.561165 7025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:42:36.565251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 6957
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:36.570379 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.561937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.0.98.62:43607)
0927 02:42:36.561999 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.562002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.562172 (+ 170us) server_negotiation.cc:407] Connection header received
0927 02:42:36.562639 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.562640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.562650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.562698 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.563326 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.563821 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.564918 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.565173 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.565267 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.567715 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.567716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.567716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.567723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.569056 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.569435 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.569436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.569437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.569490 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.569710 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.569711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.569711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.569853 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.569912 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.570164 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.570287 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:42:36.570529 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.561869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46106)
0927 02:42:36.562056 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.562094 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.562545 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.562776 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.562780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.563024 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.563222 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.563226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.563918 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.563919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.564821 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.564823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.565261 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.565262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.565384 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.565942 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.565950 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.567615 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.569130 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.569135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.569137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.569359 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.569549 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.569550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.569550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.569650 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.569917 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.569923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.570046 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.570131 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.570291 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:42:36.571153 7025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:42:36.574317 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.565743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.0.98.61:33363)
0927 02:42:36.565886 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.565891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.566017 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:42:36.566041 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.566043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.566067 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.566147 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.566915 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.567757 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.568781 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.569073 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.569140 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.571434 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.571442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.571442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.571448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.573258 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.573557 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.573560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.573560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.573615 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.573780 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.573781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.573782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.573912 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.574068 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.574137 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.574218 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:42:36.578114 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.571392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.0.1:52642)
0927 02:42:36.571432 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.571441 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.571473 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.571756 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.571757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.571883 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.571927 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.571928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.572772 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.572773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.573261 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.573263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.573610 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.573610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.573670 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.573998 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.574003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.575705 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.577142 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.577143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.577143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.577297 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.577492 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.577493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.577494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.577555 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.577803 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.577804 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.577891 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.577971 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.578051 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:42:36.578222 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.571470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.0.98.61:33363)
0927 02:42:36.571623 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.571625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.571633 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:36.571648 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.571649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.571658 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.571728 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.571999 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.572695 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.573339 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.573558 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.573612 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.575793 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.575794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.575794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.575799 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.577022 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.577359 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.577360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.577360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.577408 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.577615 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.577616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.577616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.577710 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.577758 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.577993 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.578098 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260927 02:42:36.579571 7025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Bootstrap starting.
I20260927 02:42:36.580127 7025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:36.580338 7025 log.cc:824] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:36.581135 7025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: No bootstrap required, opened a new log
I20260927 02:42:36.582461 7025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } }
I20260927 02:42:36.582579 7025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:36.582600 7025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c329716d8dd4447b1c8b9206a987619, State: Initialized, Role: FOLLOWER
I20260927 02:42:36.582713 7025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } }
I20260927 02:42:36.583000 7025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:36.582971 7032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } } }
I20260927 02:42:36.583061 7032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:42:36.584605 7043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:36.584667 7043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:36.586268 6957 master_runner.cc:428] Detected that this master 9c329716d8dd4447b1c8b9206a987619 is joining an existing cluster
I20260927 02:42:36.586309 6957 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:33363
I20260927 02:42:36.593639 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.587186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.0.98.62:43607)
0927 02:42:36.587225 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.587227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.587423 (+ 196us) server_negotiation.cc:407] Connection header received
0927 02:42:36.587469 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.587470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.587481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.587528 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.588052 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.588615 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.589645 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.589887 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.589918 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.591089 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.591090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.591091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.591097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.592553 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.592899 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.592901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.592901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.592959 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.593144 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.593144 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.593145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.593279 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.593335 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.593476 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.593546 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:42:36.593657 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.587130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46118)
0927 02:42:36.587348 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.587358 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.587419 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.587570 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.587571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.587853 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.587956 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.587959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.588683 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.588684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.589549 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.589551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.589989 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.589990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.590030 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.590491 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.590495 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.590986 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.592624 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.592625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.592625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.592838 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.593016 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.593017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.593017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.593084 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.593337 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.593338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.593382 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.593442 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.593543 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260927 02:42:36.594430 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.587292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.0.98.61:33363)
0927 02:42:36.587336 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.587338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.587428 (+ 90us) server_negotiation.cc:407] Connection header received
0927 02:42:36.587513 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.587513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.587522 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.587564 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.588036 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.588824 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.589741 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.590047 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.590116 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.591076 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.591078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.591078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.591085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.592889 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.593237 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.593238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.593238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.593289 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.593472 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.593473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.593473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.593583 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.593621 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.594263 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.594342 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:42:36.594727 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.587222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.0.1:52644)
0927 02:42:36.587376 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.587384 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.587446 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.587568 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.587569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.587848 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.587956 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.587959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.588908 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.588909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.589659 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.589670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.590129 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.590129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.590181 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.590636 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.590639 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.590997 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.592962 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.592963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.592963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.593123 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.593354 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.593355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.593355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.593414 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.593658 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.593659 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.594121 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.594544 (+ 423us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.594631 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":19,"thread_start_us":80,"threads_started":1}
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476955, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:42:36.602715 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.595804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.0.98.61:33363)
0927 02:42:36.595862 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.595864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.595997 (+ 133us) server_negotiation.cc:407] Connection header received
0927 02:42:36.596055 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.596056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.596065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.596127 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.596539 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.597278 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.598050 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.598358 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.598422 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.600421 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.600423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.600423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.600430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.601635 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.602043 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.602044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.602044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.602087 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.602267 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.602267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.602267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.602359 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.602404 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.602596 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.602648 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:42:36.602813 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.595764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.0.1:52646)
0927 02:42:36.595887 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.595936 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.596000 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.596130 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.596132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.596378 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.596475 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.596477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.597376 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.597377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.597974 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.597976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.598421 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.598422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.598472 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.598908 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.598912 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.600309 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.601714 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.601721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.601723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.601978 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.602142 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.602143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.602143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.602214 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.602414 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.602421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.602476 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.602569 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.602682 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
I20260927 02:42:36.603433 6493 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:33363: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9c329716d8dd4447b1c8b9206a987619"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 33363
}
attrs {
promote: true
}
}
dest_uuid: "6592263c8e1d49dd9be3ddc85521a8c0"
cas_config_opid_index: -1
I20260927 02:42:36.603730 6493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } }
I20260927 02:42:36.604176 7053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9c329716d8dd4447b1c8b9206a987619 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } } }
I20260927 02:42:36.604586 6999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:36.605248 7053 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:33363
I20260927 02:42:36.605470 6999 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Refusing update from remote peer 6592263c8e1d49dd9be3ddc85521a8c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 02:42:36.605491 6957 master.cc:537] Master@127.0.98.61:33363 shutting down...
I20260927 02:42:36.605530 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } } }
I20260927 02:42:36.605592 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:36.605850 7052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:42:36.606550 6480 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 02:42:36.607250 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 -> Peer 9c329716d8dd4447b1c8b9206a987619 (127.0.98.61:33363): Couldn't send request to peer 9c329716d8dd4447b1c8b9206a987619. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:42:36.608269 6957 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:36.608348 6957 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:36.608425 6957 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: stopping tablet replica
I20260927 02:42:36.621236 6957 master.cc:552] Master@127.0.98.61:33363 shutdown complete.
I20260927 02:42:36.621285 6957 master_runner.cc:305] Clearing existing system tablet
I20260927 02:42:36.621397 6957 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:36.621444 6957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:36.622061 6957 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260927 02:42:36.622505 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.622622 6957 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:36.622699 6957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.622726 6957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal
uuid: "9c329716d8dd4447b1c8b9206a987619"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "b163405a7c8458bb5bc248521511fb0c"
server_key_iv: "0b9138b81006687111845995bdc6e5ca"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.623512 6957 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:36.625098 6957 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:42:36.625166 6957 log.cc:1196] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 02:42:36.625360 6957 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Deleting consensus metadata
I20260927 02:42:36.625574 6957 master_runner.cc:315] Copying system tablet from 127.0.98.62:43607
I20260927 02:42:36.626161 6957 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43607
I20260927 02:42:36.632287 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.626516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.0.98.62:43607)
0927 02:42:36.626573 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.626574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.626621 (+ 47us) server_negotiation.cc:407] Connection header received
0927 02:42:36.626709 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.626710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.626722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.626785 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.627224 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.627757 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.628559 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.628774 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.628840 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.629820 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.629822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.629822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.629830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.631170 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.631505 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.631506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.631507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.631576 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.631760 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.631761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.631761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.631880 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.631945 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.632100 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.632183 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:42:36.632306 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.626430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46128)
0927 02:42:36.626550 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.626563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.626639 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.626802 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.626803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.626982 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.627125 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.627127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.627848 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.627849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.628470 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.628473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.628845 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.628846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.628914 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.629401 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.629406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.629723 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.631256 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.631257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.631257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.631434 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.631638 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.631639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.631639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.631701 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.631943 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.631944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.631990 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.632069 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.632184 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260927 02:42:36.632822 6513 tablet_copy_service.cc:140] P 6592263c8e1d49dd9be3ddc85521a8c0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9c329716d8dd4447b1c8b9206a987619 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46128)
I20260927 02:42:36.632916 6513 tablet_copy_service.cc:161] P 6592263c8e1d49dd9be3ddc85521a8c0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9c329716d8dd4447b1c8b9206a987619 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46128: session id = 9c329716d8dd4447b1c8b9206a987619-00000000000000000000000000000000
I20260927 02:42:36.633574 6513 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:42:36.634274 6957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:42:36.635510 6957 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: tablet copy: Starting download of 0 data blocks...
I20260927 02:42:36.635675 6957 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: tablet copy: Starting download of 1 WAL segments...
I20260927 02:42:36.636691 6957 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:42:36.637508 6513 tablet_copy_service.cc:342] P 6592263c8e1d49dd9be3ddc85521a8c0: Request end of tablet copy session 9c329716d8dd4447b1c8b9206a987619-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46128
I20260927 02:42:36.637559 6513 tablet_copy_service.cc:433] P 6592263c8e1d49dd9be3ddc85521a8c0: ending tablet copy session 9c329716d8dd4447b1c8b9206a987619-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9c329716d8dd4447b1c8b9206a987619
I20260927 02:42:36.639981 6957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:36.641209 7067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.641330 7064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.641373 7065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:36.641304 6957 server_base.cc:1034] running on GCE node
I20260927 02:42:36.641569 6957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:36.641682 6957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:36.642820 6957 hybrid_clock.cc:648] HybridClock initialized: now 1790476956642812 us; error 22 us; skew 500 ppm
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:36.644320 6957 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:42:36.644587 6957 webserver.cc:460] Webserver started at http://127.0.98.61:41851/ using document root <none> and password file <none>
I20260927 02:42:36.644692 6957 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:36.644726 6957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:36.645289 6957 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:42:36.645767 7074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.645891 6957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.645933 6957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal
uuid: "9c329716d8dd4447b1c8b9206a987619"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "b163405a7c8458bb5bc248521511fb0c"
server_key_iv: "0b9138b81006687111845995bdc6e5ca"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.645988 6957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:36.667593 6957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:36.669137 6957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:36.671938 6957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33363
I20260927 02:42:36.671955 7126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33363 every 8 connection(s)
I20260927 02:42:36.672277 6957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:42:36.673004 7127 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:42:36.673487 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Bootstrap starting.
I20260927 02:42:36.676399 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:42:36.676642 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Bootstrap complete.
I20260927 02:42:36.676843 7127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } }
I20260927 02:42:36.676915 7127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c329716d8dd4447b1c8b9206a987619, State: Initialized, Role: LEARNER
I20260927 02:42:36.676952 7127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } }
I20260927 02:42:36.677126 7127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:36.678491 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } } }
I20260927 02:42:36.678587 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:42:36.679522 7142 catalog_manager.cc:1259] Loaded cluster ID: e2e9a06fcbd5484fa0b2e57309ce8ba0
I20260927 02:42:36.679574 7142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: loading cluster ID for follower catalog manager: success
I20260927 02:42:36.680610 7142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: acquiring CA information for follower catalog manager: success
I20260927 02:42:36.680920 7142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:42:36.683297 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.676029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.0.98.61:33363)
0927 02:42:36.676160 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.676162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.676172 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:36.676194 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.676203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.676214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.676332 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.677146 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.678166 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.679231 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.679426 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.679488 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.680751 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.680753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.680753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.680761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.682074 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.682495 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.682497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.682497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.682599 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.682828 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.682829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.682829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.682934 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.683038 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.683131 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.683218 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 02:42:36.712021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39581
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43607,127.0.98.61:33363,127.0.98.60:39581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:36.793685 7145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:36.793926 7145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:36.793948 7145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:36.795220 7145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:36.795282 7145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:36.795296 7145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:36.795308 7145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:36.795320 7145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:36.797055 7145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43607,127.0.98.61:33363,127.0.98.60:39581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:36.797330 7145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:36.797576 7145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:36.802394 7151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:36.802421 7145 server_base.cc:1034] running on GCE node
W20260927 02:42:36.802390 7153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.802390 7150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:36.802918 7145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:36.803285 7145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:36.804468 7145 hybrid_clock.cc:648] HybridClock initialized: now 1790476956804452 us; error 32 us; skew 500 ppm
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:36.807014 7145 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:42:36.807482 7145 webserver.cc:460] Webserver started at http://127.0.98.60:39089/ using document root <none> and password file <none>
I20260927 02:42:36.807673 7145 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:36.807714 7145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:36.807804 7145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:36.808881 7145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data/instance:
uuid: "821b164bce744423bda40a77e03c255a"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "2c0e63c4a3543ebcfffdd681e176ed2d"
server_key_iv: "95f8e48a1542f147f545cec47bab5908"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.809194 7145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal/instance:
uuid: "821b164bce744423bda40a77e03c255a"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "2c0e63c4a3543ebcfffdd681e176ed2d"
server_key_iv: "95f8e48a1542f147f545cec47bab5908"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.810490 7145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:42:36.811244 7160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.811502 7145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:36.811611 7145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal
uuid: "821b164bce744423bda40a77e03c255a"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "2c0e63c4a3543ebcfffdd681e176ed2d"
server_key_iv: "95f8e48a1542f147f545cec47bab5908"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.811694 7145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:36.831054 7145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:36.833335 7145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:36.833461 7145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:36.838039 7145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39581
I20260927 02:42:36.838091 7212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39581 every 8 connection(s)
I20260927 02:42:36.838449 7145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:42:36.839432 7213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:36.840996 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:36.847219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 7145
I20260927 02:42:36.850165 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.841894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.0.98.62:43607)
0927 02:42:36.841961 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.841963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.841998 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:42:36.842546 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.842547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.842561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.842612 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.843224 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.843726 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.844777 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.845018 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.845074 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.847356 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.847358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.847358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.847368 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.848759 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.849213 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.849215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.849215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.849272 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.849516 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.849517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.849517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.849623 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.849685 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.849973 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.850066 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:42:36.850310 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.841689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46134)
0927 02:42:36.841875 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.841915 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.842436 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.842682 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.842686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.842932 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.843126 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.843130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.843816 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.843816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.844697 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.844700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.845086 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.845087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.845212 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.845687 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.845696 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.847251 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.848834 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.848839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.848841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.849121 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.849338 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.849339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.849339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.849458 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.849705 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.849711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.849844 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.849935 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.850092 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260927 02:42:36.850843 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:42:36.856604 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.847688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.0.98.60:39581)
0927 02:42:36.847871 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.847875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.847892 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:42:36.847909 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.847911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.847922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.847991 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.848584 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.849488 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.850166 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.850733 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.850797 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.853543 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.853550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.853550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.853556 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.855209 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.855605 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.855608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.855608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.855659 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.855867 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.855868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.855868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.856034 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.856192 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.856417 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.856492 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":117,"threads_started":1}
I20260927 02:42:36.858640 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.851102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.0.1:52668)
0927 02:42:36.851138 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.851175 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.851211 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.851365 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.851366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.851468 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.851511 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.851512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.852452 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.852453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.853246 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.853248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.853727 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.853728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.854026 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.854375 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.854379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.855761 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.857657 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.857658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.857658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.857821 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.858021 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.858022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.858022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.858076 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.858324 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.858325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.858375 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.858441 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.858533 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 02:42:36.858645 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.851161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.0.98.61:33363)
0927 02:42:36.851203 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.851205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.851213 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:36.851288 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.851289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.851300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.851364 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.851588 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.852321 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.853328 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.853607 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.853658 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.855844 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.855846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.855847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.855854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.857581 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.857893 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.857894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.857894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.857953 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.858132 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.858133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.858133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.858248 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.858318 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.858473 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.858558 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:42:36.859138 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:42:36.864761 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.859349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.0.1:60044)
0927 02:42:36.859397 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.859406 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.859440 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.859527 (+ 87us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.859528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.859660 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.859686 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.859687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.860302 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.860303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.860829 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.860830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.861142 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.861143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.861185 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.861530 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.861534 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.862590 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.863870 (+ 1280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.863870 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.863871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.864006 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.864187 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.864188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.864188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.864239 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.864452 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.864453 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.864488 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.864546 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.864641 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:42:36.864802 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.859408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.0.98.60:39581)
0927 02:42:36.859451 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.859452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.859459 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:42:36.859476 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.859477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.859485 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.859526 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.859749 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.860229 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.860895 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.861079 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.861138 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.862661 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.862663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.862663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.862669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.863799 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.864063 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.864064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.864064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.864132 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.864289 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.864289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.864290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.864380 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.864459 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.864581 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.864688 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":29}
I20260927 02:42:36.865988 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: Bootstrap starting.
I20260927 02:42:36.866761 7213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:36.867024 7213 log.cc:824] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:36.868077 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: No bootstrap required, opened a new log
I20260927 02:42:36.869544 7213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } }
I20260927 02:42:36.869680 7213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:36.869702 7213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821b164bce744423bda40a77e03c255a, State: Initialized, Role: FOLLOWER
I20260927 02:42:36.869812 7213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } }
I20260927 02:42:36.870113 7213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:36.870093 7218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } } }
I20260927 02:42:36.870167 7218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:42:36.871660 7229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:36.871712 7229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:36.874349 7145 master_runner.cc:428] Detected that this master 821b164bce744423bda40a77e03c255a is joining an existing cluster
I20260927 02:42:36.874418 7145 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39581
I20260927 02:42:36.883342 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.875915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.0.98.60:39581)
0927 02:42:36.875952 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.875954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.876237 (+ 283us) server_negotiation.cc:407] Connection header received
0927 02:42:36.876299 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.876300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.876310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.876366 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.876997 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.877789 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.879007 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.879299 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.879342 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.880858 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.880859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.880860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.880866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.882264 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.882578 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.882579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.882580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.882629 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.882806 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.882807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.882807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.882913 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.882966 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.883094 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.883242 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":22}
I20260927 02:42:36.883354 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.875915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.0.1:60056)
0927 02:42:36.876160 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.876169 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.876236 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.876466 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.876467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.876787 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.876874 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.876876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.877908 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.877909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.878917 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.878918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.879432 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.879433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.879490 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.880328 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.880332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.880758 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.882356 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.882357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.882357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.882513 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.882686 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.882686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.882687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.882751 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.882967 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.882968 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.883006 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.883066 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.883174 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":91,"thread_start_us":77,"threads_started":1}
I20260927 02:42:36.896759 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.875605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46142)
0927 02:42:36.875769 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.875780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.875840 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.876456 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.876457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.876924 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.877018 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.877021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.878014 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.878014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.878964 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.878967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.879494 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.879495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.879555 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.893039 (+ 13484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.893044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.893502 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.895542 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.895543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.895543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.895769 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.895976 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.895977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.895977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.896087 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.896355 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.896356 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.896435 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.896504 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.896625 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260927 02:42:36.896767 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.876110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.0.98.62:43607)
0927 02:42:36.876160 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.876161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.876171 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:36.876199 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.876200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.876213 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.876273 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.877114 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.877888 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.879087 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.879375 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.879417 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.893612 (+ 14195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.893614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.893615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.893624 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.895482 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.895845 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.895846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.895846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.895908 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.896159 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.896160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.896161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.896292 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.896350 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.896557 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.896665 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:42:36.896940 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.875684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.0.98.61:33363)
0927 02:42:36.876096 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.876098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.876358 (+ 260us) server_negotiation.cc:407] Connection header received
0927 02:42:36.876598 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.876600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.876611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.876665 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.877018 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.877790 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.879022 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.879315 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.879352 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.893676 (+ 14324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.893678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.893679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.893686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.895401 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.895896 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.895897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.895898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.895950 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.896188 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.896189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.896189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.896334 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.896420 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.896765 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.896845 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383}
I20260927 02:42:36.897258 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.875605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.0.1:52672)
0927 02:42:36.876125 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.876314 (+ 189us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.876366 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.876670 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.876671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.876866 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.876919 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.876920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.877908 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.877909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.878912 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.878914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.879435 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.879435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.879491 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.892932 (+ 13441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.892939 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.893506 (+ 567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.895506 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.895507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.895507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.895827 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.896024 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.896025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.896025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.896101 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.896613 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.896613 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.896650 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.897057 (+ 407us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.897150 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":102,"thread_start_us":197,"threads_started":1}
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476955, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:42:36.906313 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.899319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.0.1:60060)
0927 02:42:36.899362 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.899372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.899430 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.899558 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.899559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.899729 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.899780 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.899782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.900483 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.900484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.901330 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.901332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.901698 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.901699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.901763 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.902116 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.902121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.903531 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.905246 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.905247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.905247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.905414 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.905635 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.905637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.905637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.905707 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.905986 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.905987 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.906085 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.906152 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.906242 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:42:36.906360 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.899373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.0.98.60:39581)
0927 02:42:36.899407 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.899409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.899417 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:36.899487 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.899488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.899495 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.899541 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.899885 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.900349 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.901428 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.901623 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.901688 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.903636 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.903637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.903637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.903644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.905110 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.905485 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.905486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.905486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.905543 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.905774 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.905775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.905775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.905903 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.905968 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.906189 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.906262 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:42:36.907262 6493 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39581: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "821b164bce744423bda40a77e03c255a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39581
}
attrs {
promote: true
}
}
dest_uuid: "6592263c8e1d49dd9be3ddc85521a8c0"
cas_config_opid_index: 5
I20260927 02:42:36.907505 6493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } }
I20260927 02:42:36.907821 7052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 821b164bce744423bda40a77e03c255a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.908077 7187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:36.908895 7187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Refusing update from remote peer 6592263c8e1d49dd9be3ddc85521a8c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:42:36.908926 7052 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39581
I20260927 02:42:36.908999 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.909073 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:36.909402 7145 master.cc:537] Master@127.0.98.60:39581 shutting down...
I20260927 02:42:36.909403 7052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:42:36.910617 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 -> Peer 821b164bce744423bda40a77e03c255a (127.0.98.60:39581): Couldn't send request to peer 821b164bce744423bda40a77e03c255a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:42:36.912758 7145 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:36.912873 7145 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:36.912928 7145 tablet_replica.cc:330] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: stopping tablet replica
I20260927 02:42:36.913719 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.909246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.0.1:52674)
0927 02:42:36.909624 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.909635 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.909675 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.909960 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.909961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.909963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:36.910009 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.910011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.910964 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.910965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.912352 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.912357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.913425 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.913426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.913500 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.913528 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.913578 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.913640 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359}
I20260927 02:42:36.914052 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.909365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.0.98.61:33363)
0927 02:42:36.909818 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.909820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.909829 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:36.909852 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.909854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.909877 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.909938 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:36.910097 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.910806 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.912492 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.913117 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.913190 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.913271 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.913934 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.913986 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426}
W20260927 02:42:36.914510 6446 connection.cc:535] client connection to 127.0.98.60:39581 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:42:36.914708 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 LEARNER]: Refusing update from remote peer 6592263c8e1d49dd9be3ddc85521a8c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:42:36.915215 7052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 02:42:36.915835 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 821b164bce744423bda40a77e03c255a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.916935 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.917037 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:42:36.917265 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6592263c8e1d49dd9be3ddc85521a8c0. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: true } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.917325 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:42:36.917553 7053 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: attempting to promote NON_VOTER 9c329716d8dd4447b1c8b9206a987619 to VOTER
I20260927 02:42:36.917795 7053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } }
I20260927 02:42:36.918350 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.918239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.0.1:60066)
0927 02:42:36.918273 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.918318 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 02:42:36.918421 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 LEARNER]: Refusing update from remote peer 6592263c8e1d49dd9be3ddc85521a8c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 02:42:36.918668 7240 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 02:42:36.918807 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.918704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.0.1:60074)
0927 02:42:36.918754 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.918781 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:42:36.918869 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 -> Peer 821b164bce744423bda40a77e03c255a (127.0.98.60:39581): Couldn't send request to peer 821b164bce744423bda40a77e03c255a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39581: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:42:36.919354 7052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9c329716d8dd4447b1c8b9206a987619 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.919521 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9c329716d8dd4447b1c8b9206a987619 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.920284 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.920359 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:36.920370 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.920477 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:36.920537 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.920610 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:42:36.926262 7145 master.cc:552] Master@127.0.98.60:39581 shutdown complete.
I20260927 02:42:36.926311 7145 master_runner.cc:305] Clearing existing system tablet
I20260927 02:42:36.926419 7145 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:36.926462 7145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:36.927124 7145 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:36.927630 7242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.927758 7145 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:36.927815 7145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.927832 7145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal
uuid: "821b164bce744423bda40a77e03c255a"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "2c0e63c4a3543ebcfffdd681e176ed2d"
server_key_iv: "95f8e48a1542f147f545cec47bab5908"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.928531 7145 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:36.929926 7145 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:42:36.929991 7145 log.cc:1196] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 02:42:36.930219 7145 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: Deleting consensus metadata
I20260927 02:42:36.930323 7145 master_runner.cc:315] Copying system tablet from 127.0.98.62:43607
I20260927 02:42:36.930877 7145 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43607
I20260927 02:42:36.937214 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.931249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.0.98.62:43607)
0927 02:42:36.931303 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:36.931306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:36.931363 (+ 57us) server_negotiation.cc:407] Connection header received
0927 02:42:36.931434 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:36.931434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:36.931444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:36.931491 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:36.931818 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.932315 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.933092 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.933301 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.933362 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.934644 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:36.934645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:36.934646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:36.934653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:36.936073 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.936487 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.936488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.936489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.936545 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:36.936721 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:36.936722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:36.936722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:36.936834 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:36.936891 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:36.937032 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 02:42:36.937109 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:42:36.937269 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.931176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46148)
0927 02:42:36.931292 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.931305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:36.931382 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:36.931496 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:36.931496 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:36.931687 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:36.931729 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.931731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.932418 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.932419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.933003 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:36.933006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:36.933374 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:36.933375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:36.933436 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:36.934211 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:36.934216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:36.934514 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:36.936209 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.936211 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.936212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.936408 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.936606 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:36.936606 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:36.936607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:36.936661 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:36.936892 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:36.936893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:36.936941 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 02:42:36.936999 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:42:36.937110 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260927 02:42:36.937932 6513 tablet_copy_service.cc:140] P 6592263c8e1d49dd9be3ddc85521a8c0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 821b164bce744423bda40a77e03c255a ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46148)
I20260927 02:42:36.938007 6513 tablet_copy_service.cc:161] P 6592263c8e1d49dd9be3ddc85521a8c0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 821b164bce744423bda40a77e03c255a at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46148: session id = 821b164bce744423bda40a77e03c255a-00000000000000000000000000000000
I20260927 02:42:36.938580 6513 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:42:36.939288 7145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:42:36.940315 7145 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: tablet copy: Starting download of 0 data blocks...
I20260927 02:42:36.940487 7145 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: tablet copy: Starting download of 1 WAL segments...
I20260927 02:42:36.941387 7145 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:42:36.942219 6513 tablet_copy_service.cc:342] P 6592263c8e1d49dd9be3ddc85521a8c0: Request end of tablet copy session 821b164bce744423bda40a77e03c255a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46148
I20260927 02:42:36.942286 6513 tablet_copy_service.cc:433] P 6592263c8e1d49dd9be3ddc85521a8c0: ending tablet copy session 821b164bce744423bda40a77e03c255a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 821b164bce744423bda40a77e03c255a
I20260927 02:42:36.944656 7145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:36.945889 7249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.945933 7252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:36.945894 7250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:36.946713 7145 server_base.cc:1034] running on GCE node
I20260927 02:42:36.946813 7145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:36.946965 7145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:36.948117 7145 hybrid_clock.cc:648] HybridClock initialized: now 1790476956948106 us; error 27 us; skew 500 ppm
Sep 27 02:42:36 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:36.949462 7145 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:42:36.949738 7145 webserver.cc:460] Webserver started at http://127.0.98.60:33535/ using document root <none> and password file <none>
I20260927 02:42:36.949854 7145 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:36.949887 7145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:36.950603 7145 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:36.951189 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:36.951328 7145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:36.951370 7145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal
uuid: "821b164bce744423bda40a77e03c255a"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "2c0e63c4a3543ebcfffdd681e176ed2d"
server_key_iv: "95f8e48a1542f147f545cec47bab5908"
server_key_version: "encryptionkey@0"
I20260927 02:42:36.951426 7145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:36.967182 7145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:36.968978 7145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:36.971427 7145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39581
I20260927 02:42:36.971710 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39581 every 8 connection(s)
I20260927 02:42:36.971800 7145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:42:36.972823 7312 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:42:36.973392 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: Bootstrap starting.
I20260927 02:42:36.976997 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:42:36.977290 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: Bootstrap complete.
I20260927 02:42:36.977492 7312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } }
I20260927 02:42:36.977595 7312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 821b164bce744423bda40a77e03c255a, State: Initialized, Role: LEARNER
I20260927 02:42:36.977640 7312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } }
I20260927 02:42:36.977856 7312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:36.977844 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:36.977953 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: This master's current role is: LEARNER
I20260927 02:42:36.979395 7326 catalog_manager.cc:1259] Loaded cluster ID: e2e9a06fcbd5484fa0b2e57309ce8ba0
I20260927 02:42:36.979445 7326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: loading cluster ID for follower catalog manager: success
I20260927 02:42:36.980464 7326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: acquiring CA information for follower catalog manager: success
I20260927 02:42:36.980747 7326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:42:37.022703 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.016377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.0.98.60:39581)
0927 02:42:37.016563 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.016566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.016574 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:37.016601 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.016601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.016612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.016683 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.017234 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.018006 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.018677 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.018863 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.018916 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.020185 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.020186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.020187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.020195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.021658 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.021985 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.021986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.021987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.022049 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.022231 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.022232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.022232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.022338 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.022432 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.022526 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.022603 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
I20260927 02:42:37.023342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 6555
I20260927 02:42:37.027635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44895
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43607
--webserver_port=36291
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43607,127.0.98.61:33363
--tserver_master_addrs=127.0.98.62:43607,127.0.98.61:33363,127.0.98.60:39581 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:37.111152 7328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:37.111380 7328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:37.111405 7328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:37.112586 7328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:37.112638 7328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:37.112665 7328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:37.114185 7328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36291
--tserver_master_addrs=127.0.98.62:43607,127.0.98.61:33363,127.0.98.60:39581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:37.114440 7328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:37.114693 7328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:37.119891 7334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:37.120007 7328 server_base.cc:1034] running on GCE node
W20260927 02:42:37.119902 7333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:37.120466 7336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:37.120690 7328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:37.120970 7328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:37.122151 7328 hybrid_clock.cc:648] HybridClock initialized: now 1790476957122132 us; error 30 us; skew 500 ppm
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:37.124821 7328 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:42:37.125244 7328 webserver.cc:460] Webserver started at http://127.0.98.1:36291/ using document root <none> and password file <none>
I20260927 02:42:37.125463 7328 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:37.125525 7328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:37.127012 7328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:37.127736 7343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.127940 7328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:37.128028 7328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "5db7bf4890bd47d38125dd42e242de88"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "b7711e9827be10d553c6f874a6027fe4"
server_key_iv: "30c68f868ee3c2cf1134d1bd437f5753"
server_key_version: "encryptionkey@0"
I20260927 02:42:37.128403 7328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:37.171371 7328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:37.173590 7328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:37.173753 7328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:37.173987 7328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:37.174288 7328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:37.174320 7328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.174350 7328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:37.174367 7328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.180464 7328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44895
I20260927 02:42:37.180497 7456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44895 every 8 connection(s)
I20260927 02:42:37.180850 7328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:37.182914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 7328
I20260927 02:42:37.183072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 6689
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:37.189728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40497
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43607
--webserver_port=35347
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43607,127.0.98.61:33363
--tserver_master_addrs=127.0.98.62:43607,127.0.98.61:33363,127.0.98.60:39581 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:42:37.192790 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.182023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44871 (local address 127.0.98.60:39581)
0927 02:42:37.182072 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.182074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.182083 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:37.182428 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.182430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.182444 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.182493 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.183434 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.184150 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.185732 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.186033 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.186071 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.189554 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.189555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.189556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.189565 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.191029 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.191611 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.191612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.191612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.191682 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.191971 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.191972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.191972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.192110 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.192151 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.192593 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.192686 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:42:37.192790 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.181543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36171 (local address 127.0.98.62:43607)
0927 02:42:37.181637 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.181640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.181860 (+ 220us) server_negotiation.cc:407] Connection header received
0927 02:42:37.182441 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.182442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.182453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.182483 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.183421 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.184160 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.185721 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.186027 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.186065 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.189751 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.189752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.189753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.189761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.191156 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.191611 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.191612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.191612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.191682 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.191971 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.191972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.191972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.192110 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.192151 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.192593 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.192686 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:42:37.193410 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.181557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.1:36171)
0927 02:42:37.181741 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.181784 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.182340 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.182908 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.182909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.183170 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.183331 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.183336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.184274 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.184274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.185620 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.185622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.186154 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.186155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.186298 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.187054 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.187057 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.189641 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.191234 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.191235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.191235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.191524 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.191760 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.191760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.191761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.191895 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.192196 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.192202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.192415 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.192895 (+ 480us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.193123 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":101,"thread_start_us":103,"threads_started":1}
I20260927 02:42:37.193413 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.181556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.98.1:44871)
0927 02:42:37.181741 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.181783 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.182334 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.182564 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.182568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.183055 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.183332 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.183337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.184275 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.184276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.185621 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.185624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.186151 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.186152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.186298 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.186936 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.186946 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.189392 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.191129 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.191140 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.191143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.191524 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.191746 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.191746 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.191747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.191895 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.192207 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.192208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.192415 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.192915 (+ 500us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.193195 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":170,"thread_start_us":94,"threads_started":1}
I20260927 02:42:37.193621 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.181956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37881 (local address 127.0.98.61:33363)
0927 02:42:37.181995 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.181997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.182073 (+ 76us) server_negotiation.cc:407] Connection header received
0927 02:42:37.182440 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.182442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.182453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.182483 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.183752 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.184470 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.185721 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.186017 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.186065 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.190790 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.190792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.190792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.190800 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.192332 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.192752 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.192753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.192753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.192811 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.193064 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.193065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.193065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.193204 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.193241 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.193416 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.193498 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:42:37.193886 7457 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39581
I20260927 02:42:37.193977 7457 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.194082 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.181885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.98.1:37881)
0927 02:42:37.182011 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.182020 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.182348 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.182579 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.182580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.183610 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.183671 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.183672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.184548 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.184549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.185620 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.185622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.186153 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.186154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.186298 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.186958 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.186963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.190686 (+ 3723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.192451 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.192452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.192452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.192670 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.192940 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.192941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.192941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.193001 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.193280 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.193281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.193324 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.193700 (+ 376us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.193803 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":13,"thread_start_us":73,"threads_started":1}
I20260927 02:42:37.194128 7457 heartbeater.cc:511] Master 127.0.98.60:39581 requested a full tablet report, sending...
I20260927 02:42:37.194226 7458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43607
I20260927 02:42:37.194268 7458 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.194334 7458 heartbeater.cc:511] Master 127.0.98.62:43607 requested a full tablet report, sending...
I20260927 02:42:37.194558 7459 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33363
I20260927 02:42:37.194566 6493 ts_manager.cc:194] Re-registered known tserver with Master: 5db7bf4890bd47d38125dd42e242de88 (127.0.98.1:44895)
I20260927 02:42:37.194600 7459 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.194607 7276 ts_manager.cc:194] Registered new tserver with Master: 5db7bf4890bd47d38125dd42e242de88 (127.0.98.1:44895)
I20260927 02:42:37.194656 7459 heartbeater.cc:511] Master 127.0.98.61:33363 requested a full tablet report, sending...
I20260927 02:42:37.195001 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36171
I20260927 02:42:37.195003 7091 ts_manager.cc:194] Registered new tserver with Master: 5db7bf4890bd47d38125dd42e242de88 (127.0.98.1:44895)
W20260927 02:42:37.272228 7464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:37.272521 7464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:37.272560 7464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:37.273804 7464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:37.273856 7464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:37.273895 7464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:37.275473 7464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35347
--tserver_master_addrs=127.0.98.62:43607,127.0.98.61:33363,127.0.98.60:39581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:37.275731 7464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:37.275959 7464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:37.281282 7470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:37.281271 7472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:37.281271 7469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:37.281556 7464 server_base.cc:1034] running on GCE node
I20260927 02:42:37.281766 7464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:37.281967 7464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:37.283149 7464 hybrid_clock.cc:648] HybridClock initialized: now 1790476957283126 us; error 32 us; skew 500 ppm
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:37.285815 7464 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:42:37.286245 7464 webserver.cc:460] Webserver started at http://127.0.98.2:35347/ using document root <none> and password file <none>
I20260927 02:42:37.286438 7464 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:37.286497 7464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:37.287838 7464 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:37.288517 7479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.288727 7464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:42:37.288816 7464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "d363b47e300a45abaab6399f1182ea65"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "5bb4425c6fee31e396f80d1a781c0c30"
server_key_iv: "8acb7949179f4ac09a863ad6e860ad7a"
server_key_version: "encryptionkey@0"
I20260927 02:42:37.289160 7464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:37.314630 7464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:37.316977 7464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:37.317154 7464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:37.317474 7464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:37.317814 7464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:37.317848 7464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.317869 7464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:37.317893 7464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.323447 7464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40497
I20260927 02:42:37.323493 7592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40497 every 8 connection(s)
I20260927 02:42:37.323837 7464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:37.327715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 7464
I20260927 02:42:37.327876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 6823
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:42:37.336186 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.324860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38449 (local address 127.0.98.62:43607)
0927 02:42:37.324900 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.324902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.325045 (+ 143us) server_negotiation.cc:407] Connection header received
0927 02:42:37.325964 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.325966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.325979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.326030 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.326906 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.327669 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.329289 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.329578 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.329620 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.332902 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.332903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.332903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.332911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.334826 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335227 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.335229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.335229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.335284 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335485 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.335486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.335486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.335668 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.335747 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.336013 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.336094 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:42:37.336207 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.324860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56685 (local address 127.0.98.61:33363)
0927 02:42:37.324903 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.324905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.325092 (+ 187us) server_negotiation.cc:407] Connection header received
0927 02:42:37.325974 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.325976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.325987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.326018 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.326932 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.327707 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.329425 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.329726 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.329762 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.333441 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.333443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.333443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.333451 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.335135 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335463 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.335464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.335464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.335516 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335695 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.335709 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.335710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.335829 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.335869 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.336048 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.336116 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:42:37.336215 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.324859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58697 (local address 127.0.98.60:39581)
0927 02:42:37.324898 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.324900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.324912 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:42:37.325975 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.325976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.325988 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.326018 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.327834 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.328480 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.329425 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.329717 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.329755 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.332533 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.332535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.332535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.332542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.334619 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335145 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.335146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.335146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.335218 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335480 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.335481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.335481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.335616 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.335657 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.336025 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.336109 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:42:37.336426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43277
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43607
--webserver_port=36843
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43607,127.0.98.61:33363
--tserver_master_addrs=127.0.98.62:43607,127.0.98.61:33363,127.0.98.60:39581 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:42:37.336773 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.324613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.98.2:56685)
0927 02:42:37.325053 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.325059 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.325866 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.326105 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.326107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.326606 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.326820 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.326825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.328453 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.328454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.329337 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.329338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.329832 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.329833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.329898 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.330456 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.330466 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.333340 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.335207 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.335208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.335385 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.335580 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.335581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.335629 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.335912 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.335913 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.335946 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.336330 (+ 384us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.336535 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":36,"thread_start_us":75,"threads_started":1}
I20260927 02:42:37.337116 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.324589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.98.2:58697)
0927 02:42:37.324780 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.324824 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.325853 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.326104 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.326107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.326592 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.326820 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.326825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.328565 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.328566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.329331 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.329333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.329831 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.329832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.329898 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.330486 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.330490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.332425 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.334706 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.334715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.334717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.335058 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.335285 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.335286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.335412 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.335699 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.335706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.335877 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.336664 (+ 787us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.336826 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":78,"thread_start_us":107,"threads_started":1}
I20260927 02:42:37.337257 7595 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33363
I20260927 02:42:37.337371 7595 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.337381 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.324589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:38449)
0927 02:42:37.324962 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.324971 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.325862 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.326103 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.326107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.326587 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.326818 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.326823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.327851 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.327852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.329186 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.329189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.329704 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.329705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.329841 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.330533 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.330538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.332809 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.334916 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.334917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.334917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.335149 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.335346 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.335346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.335347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.335422 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.335798 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.335798 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.335877 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.336639 (+ 762us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.336765 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":23,"thread_start_us":88,"threads_started":1}
I20260927 02:42:37.337534 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39581
I20260927 02:42:37.337571 7593 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.337669 7593 heartbeater.cc:511] Master 127.0.98.60:39581 requested a full tablet report, sending...
I20260927 02:42:37.337845 7595 heartbeater.cc:511] Master 127.0.98.61:33363 requested a full tablet report, sending...
I20260927 02:42:37.338060 7594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43607
I20260927 02:42:37.338115 7594 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.338146 7091 ts_manager.cc:194] Registered new tserver with Master: d363b47e300a45abaab6399f1182ea65 (127.0.98.2:40497)
I20260927 02:42:37.338172 7594 heartbeater.cc:511] Master 127.0.98.62:43607 requested a full tablet report, sending...
I20260927 02:42:37.338475 7276 ts_manager.cc:194] Registered new tserver with Master: d363b47e300a45abaab6399f1182ea65 (127.0.98.2:40497)
I20260927 02:42:37.338617 6493 ts_manager.cc:194] Re-registered known tserver with Master: d363b47e300a45abaab6399f1182ea65 (127.0.98.2:40497)
I20260927 02:42:37.339047 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38449
I20260927 02:42:37.367059 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.364481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.0.98.60:39581)
0927 02:42:37.364540 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.364542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.364552 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:37.364613 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.364613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.364637 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.364695 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:37.364832 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.365430 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.366358 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.366738 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.366799 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.366871 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.366948 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.366981 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:42:37.367059 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.364367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.0.1:60092)
0927 02:42:37.364466 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.364479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.364546 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.364710 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.364711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.364712 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:37.364762 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.364764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.365512 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.365513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.366271 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.366273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.366818 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.366819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.366879 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.366884 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.366932 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.366981 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 02:42:37.367483 7286 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 02:42:37.367628 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6592263c8e1d49dd9be3ddc85521a8c0. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:37.367727 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: This master's current role is: LEARNER
W20260927 02:42:37.419375 7600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:37.419602 7600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:37.419636 7600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:37.421147 7600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:37.421211 7600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:37.421252 7600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:37.422865 7600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36843
--tserver_master_addrs=127.0.98.62:43607,127.0.98.61:33363,127.0.98.60:39581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:37.423130 7600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:37.423365 7600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:37.428529 7605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:37.428648 7600 server_base.cc:1034] running on GCE node
W20260927 02:42:37.428560 7606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:37.429740 7608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:37.429970 7600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:37.430222 7600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:37.431382 7600 hybrid_clock.cc:648] HybridClock initialized: now 1790476957431368 us; error 27 us; skew 500 ppm
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:37.434048 7600 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:42:37.434548 7600 webserver.cc:460] Webserver started at http://127.0.98.3:36843/ using document root <none> and password file <none>
I20260927 02:42:37.434791 7600 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:37.434857 7600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:37.436487 7600 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:42:37.437266 7615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.437563 7600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:37.437665 7600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "bd77f6002a0941d7acd2243f0b8f78fa"
format_stamp: "Formatted at 2026-09-27 02:42:36 on dist-test-slave-44kr"
server_key: "556e2c0dd5d6c0f80e059036cd0bb7a6"
server_key_iv: "849b52431614c6d9c168793ada1dcd0f"
server_key_version: "encryptionkey@0"
I20260927 02:42:37.438053 7600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:37.448872 7600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:37.451246 7600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:37.451395 7600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:37.451634 7600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:37.451938 7600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:37.451975 7600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.452004 7600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:37.452028 7600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.457840 7600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43277
I20260927 02:42:37.457890 7728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43277 every 8 connection(s)
I20260927 02:42:37.458240 7600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:37.461849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 7600
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:42:37.471909 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.458989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.3:59919)
0927 02:42:37.459291 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.459333 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.459964 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.460325 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.460330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.460810 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.461030 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.461035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.462827 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.462828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.463611 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.463612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.464098 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.464099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.464169 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.464747 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.464764 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.467083 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.470266 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.470279 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.470282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.470591 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.470801 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.470801 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.470802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.470925 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.471163 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.471171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.471338 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.471452 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.471636 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":27,"thread_start_us":88,"threads_started":1}
I20260927 02:42:37.472122 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.459224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38399 (local address 127.0.98.61:33363)
0927 02:42:37.459270 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.459272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.459430 (+ 158us) server_negotiation.cc:407] Connection header received
0927 02:42:37.460236 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.460238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.460248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.460310 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.461143 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.461906 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.463361 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.463649 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.463720 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.468799 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.468801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.468801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.468808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.470668 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.471042 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.471043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.471044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.471096 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.471294 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.471296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.471296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.471431 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.471485 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.471966 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.472040 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 02:42:37 dist-test-slave-44kr krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:42:37.473081 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.459086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59919 (local address 127.0.98.62:43607)
0927 02:42:37.459188 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.459190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.459436 (+ 246us) server_negotiation.cc:407] Connection header received
0927 02:42:37.460102 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.460104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.460119 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.460178 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.462006 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.462719 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.463694 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.463983 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.464022 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.467189 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.467191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.467191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.467199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.470109 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.470678 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.470679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.470680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.470739 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.470986 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.470987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.470988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.471100 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.471156 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.472904 (+ 1748us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.472988 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 02:42:37.473215 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.463812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.0.98.62:43607)
0927 02:42:37.464711 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.464713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.464875 (+ 162us) server_negotiation.cc:407] Connection header received
0927 02:42:37.464897 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.464897 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.464911 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.464974 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.465691 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.466528 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.467598 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.467915 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.467973 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.469521 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.469522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.469523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.469530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.471352 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.471946 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.471947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.471947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.472004 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.472414 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.472415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.472415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.472544 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.472650 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.472998 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.473077 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":57,"threads_started":1}
I20260927 02:42:37.471911 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.458989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.98.3:38399)
0927 02:42:37.459289 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.459333 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.459972 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.460343 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.460343 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.460816 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.461030 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.461035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.462051 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.462052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.463265 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.463268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.463785 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.463786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.463930 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.464864 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.464869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.468699 (+ 3830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.470760 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.470761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.470761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.470964 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.471176 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.471176 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.471176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.471229 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.471511 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.471512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.471547 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.471600 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.471700 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":58,"thread_start_us":76,"threads_started":1}
I20260927 02:42:37.473742 7731 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33363
I20260927 02:42:37.473855 7731 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.474084 7731 heartbeater.cc:511] Master 127.0.98.61:33363 requested a full tablet report, sending...
I20260927 02:42:37.473750 7730 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43607
I20260927 02:42:37.474321 7730 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.474378 7730 heartbeater.cc:511] Master 127.0.98.62:43607 requested a full tablet report, sending...
I20260927 02:42:37.474609 7091 ts_manager.cc:194] Registered new tserver with Master: bd77f6002a0941d7acd2243f0b8f78fa (127.0.98.3:43277)
I20260927 02:42:37.474642 6493 ts_manager.cc:194] Re-registered known tserver with Master: bd77f6002a0941d7acd2243f0b8f78fa (127.0.98.3:43277)
I20260927 02:42:37.475005 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59919
I20260927 02:42:37.474958 6492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:42:37.477240 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.464298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.0.98.61:33363)
0927 02:42:37.464437 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.464439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.464971 (+ 532us) server_negotiation.cc:407] Connection header received
0927 02:42:37.465082 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.465083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.465095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.465130 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.465837 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.466683 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.467664 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.468001 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.468043 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.470267 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.470268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.470269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.470277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.473070 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.474114 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.474115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.474116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.474260 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.476591 (+ 2331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.476592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.476592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.476739 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.476814 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.477007 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.477098 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":51,"threads_started":1}
W20260927 02:42:37.477386 6492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:37.478068 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.463585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.0.98.60:39581)
0927 02:42:37.465715 (+ 2130us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.465717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.465858 (+ 141us) server_negotiation.cc:407] Connection header received
0927 02:42:37.465879 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.465880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.465891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.465946 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.466690 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.467444 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.468372 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.468805 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.468890 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.473900 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.473903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.473903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.473912 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.475964 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.476350 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.476351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.476351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.476449 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.476914 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.476915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.476915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.477042 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.477127 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.477889 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.477969 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2107,"thread_start_us":50,"threads_started":1}
I20260927 02:42:37.478262 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: true } } }
I20260927 02:42:37.478351 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:37.478854 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.459141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44089 (local address 127.0.98.60:39581)
0927 02:42:37.459222 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.459224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.459899 (+ 675us) server_negotiation.cc:407] Connection header received
0927 02:42:37.460172 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.460174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.460188 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.460241 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.461128 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.461930 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.463374 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.463667 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.463705 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.475772 (+ 12067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.475774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.475774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.475782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.477318 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.477984 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.477984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.477985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.478036 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.478270 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.478271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.478271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.478380 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.478418 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.478701 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.478772 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 02:42:37.478885 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.458989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.98.3:44089)
0927 02:42:37.459786 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.459796 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.460034 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.460655 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.460656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.460833 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.461029 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.461034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.462050 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.462051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.463281 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.463284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.463785 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.463786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.463929 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.464590 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.464600 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.475665 (+ 11065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.477662 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.477663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.477664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.477878 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.478140 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.478141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.478141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.478208 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.478531 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.478532 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.478593 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.478665 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.478781 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":71,"thread_start_us":87,"threads_started":1}
I20260927 02:42:37.479260 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39581
I20260927 02:42:37.479315 7729 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.479399 7729 heartbeater.cc:511] Master 127.0.98.60:39581 requested a full tablet report, sending...
I20260927 02:42:37.479645 7276 ts_manager.cc:194] Registered new tserver with Master: bd77f6002a0941d7acd2243f0b8f78fa (127.0.98.3:43277)
I20260927 02:42:37.480165 7053 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: attempting to promote NON_VOTER 821b164bce744423bda40a77e03c255a to VOTER
I20260927 02:42:37.480291 7053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } }
I20260927 02:42:37.480815 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Refusing update from remote peer 6592263c8e1d49dd9be3ddc85521a8c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260927 02:42:37.480827 7286 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 LEARNER]: Refusing update from remote peer 6592263c8e1d49dd9be3ddc85521a8c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 02:42:37.480988 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:42:37.481102 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:42:37.481611 7053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 821b164bce744423bda40a77e03c255a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:37.481750 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 821b164bce744423bda40a77e03c255a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:37.482394 7286 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 821b164bce744423bda40a77e03c255a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:37.520465 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:37.520612 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:37.520586 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:37.520696 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:42:37.520665 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:37.520781 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:37.520665 7754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:37.520825 7754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:37.521533 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:37.521656 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:42:37.526439 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.521870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43277 (local address 127.0.0.1:47876)
0927 02:42:37.522127 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.522139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.522192 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.522503 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.522504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.522506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:37.522569 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.522571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.523876 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.523877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.525099 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.525102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.526067 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.526068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.526220 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.526226 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.526285 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.526343 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":102,"threads_started":1}
I20260927 02:42:37.526533 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.521759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40497 (local address 127.0.0.1:38834)
0927 02:42:37.522127 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.522139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.522192 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.522563 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.522564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.522565 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:37.522591 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.522592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.524024 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.524025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.525121 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.525123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.526299 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.526300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.526355 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.526358 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.526384 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.526419 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318}
I20260927 02:42:37.526580 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.521968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.0.98.3:43277)
0927 02:42:37.522326 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.522331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.522342 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:37.522387 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.522388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.522405 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.522655 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:37.522715 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.523776 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.525182 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.525976 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.526039 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.526412 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.526443 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.526489 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":69,"threads_started":1}
I20260927 02:42:37.526798 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.522235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.0.98.1:44895)
0927 02:42:37.522402 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.522407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.522417 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:37.522456 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.522458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.522472 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.522562 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:37.522750 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.524201 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.525150 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.526158 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.526218 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.526288 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.526670 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.526715 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260927 02:42:37.526954 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.522064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44895 (local address 127.0.0.1:50768)
0927 02:42:37.522278 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.522286 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.522320 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.522628 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.522629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.522631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:37.522679 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.522681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.524289 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.524290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.525057 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.525059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.526333 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.526334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.526406 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.526412 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.526458 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.526501 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":53,"threads_started":1}
I20260927 02:42:37.526947 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.521912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.0.98.2:40497)
0927 02:42:37.522423 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.522426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.522437 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:37.522470 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.522471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.522484 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.522660 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:37.522714 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.523944 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.525352 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.526193 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.526478 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.526793 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.526822 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.526869 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":99,"threads_started":1}
I20260927 02:42:37.527647 7391 tablet_service.cc:1467] Processing CreateTablet for tablet 98e456dcbc59453c94bdd51d16398bca (DEFAULT_TABLE table=test-table [id=6b7e9ef105cb4ecbab7abc4febcdd161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:37.527681 7527 tablet_service.cc:1467] Processing CreateTablet for tablet 98e456dcbc59453c94bdd51d16398bca (DEFAULT_TABLE table=test-table [id=6b7e9ef105cb4ecbab7abc4febcdd161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:37.528016 7391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e456dcbc59453c94bdd51d16398bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:37.528012 7527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e456dcbc59453c94bdd51d16398bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:37.528240 7663 tablet_service.cc:1467] Processing CreateTablet for tablet 98e456dcbc59453c94bdd51d16398bca (DEFAULT_TABLE table=test-table [id=6b7e9ef105cb4ecbab7abc4febcdd161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:37.528573 7663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e456dcbc59453c94bdd51d16398bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:37.530987 7762 tablet_bootstrap.cc:492] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: Bootstrap starting.
I20260927 02:42:37.531667 7762 tablet_bootstrap.cc:654] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:37.531976 7762 log.cc:824] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:37.532174 7763 tablet_bootstrap.cc:492] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: Bootstrap starting.
I20260927 02:42:37.532464 7765 tablet_bootstrap.cc:492] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: Bootstrap starting.
I20260927 02:42:37.532939 7762 tablet_bootstrap.cc:492] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: No bootstrap required, opened a new log
I20260927 02:42:37.532946 7763 tablet_bootstrap.cc:654] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:37.533013 7762 ts_tablet_manager.cc:1397] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:37.533085 7765 tablet_bootstrap.cc:654] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:37.533236 7763 log.cc:824] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:37.533393 7765 log.cc:824] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:37.534017 7763 tablet_bootstrap.cc:492] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: No bootstrap required, opened a new log
I20260927 02:42:37.534200 7765 tablet_bootstrap.cc:492] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: No bootstrap required, opened a new log
I20260927 02:42:37.534220 7763 ts_tablet_manager.cc:1397] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:37.534265 7765 ts_tablet_manager.cc:1397] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:37.535682 7762 raft_consensus.cc:348] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } }
I20260927 02:42:37.535880 7762 raft_consensus.cc:374] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:37.535921 7762 raft_consensus.cc:729] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db7bf4890bd47d38125dd42e242de88, State: Initialized, Role: FOLLOWER
I20260927 02:42:37.536090 7762 consensus_queue.cc:260] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } }
I20260927 02:42:37.536422 7762 ts_tablet_manager.cc:1428] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:37.536332 7763 raft_consensus.cc:348] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } }
I20260927 02:42:37.536520 7458 heartbeater.cc:503] Master 127.0.98.62:43607 was elected leader, sending a full tablet report...
I20260927 02:42:37.536526 7763 raft_consensus.cc:374] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:37.536561 7763 raft_consensus.cc:729] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd77f6002a0941d7acd2243f0b8f78fa, State: Initialized, Role: FOLLOWER
I20260927 02:42:37.536636 7765 raft_consensus.cc:348] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } }
I20260927 02:42:37.536791 7765 raft_consensus.cc:374] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:37.536825 7765 raft_consensus.cc:729] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d363b47e300a45abaab6399f1182ea65, State: Initialized, Role: FOLLOWER
I20260927 02:42:37.536926 7765 consensus_queue.cc:260] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } }
I20260927 02:42:37.537168 7765 ts_tablet_manager.cc:1428] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 02:42:37.537376 7763 consensus_queue.cc:260] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } }
I20260927 02:42:37.537534 7594 heartbeater.cc:503] Master 127.0.98.62:43607 was elected leader, sending a full tablet report...
I20260927 02:42:37.537676 7763 ts_tablet_manager.cc:1428] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:37.537860 7730 heartbeater.cc:503] Master 127.0.98.62:43607 was elected leader, sending a full tablet report...
W20260927 02:42:37.574798 7596 tablet.cc:2369] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:42:37.682581 7460 tablet.cc:2369] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:42:37.709304 7732 tablet.cc:2369] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:37.944604 7768 raft_consensus.cc:482] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:37.944751 7768 raft_consensus.cc:504] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } }
I20260927 02:42:37.945256 7768 leader_election.cc:288] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d363b47e300a45abaab6399f1182ea65 (127.0.98.2:40497), bd77f6002a0941d7acd2243f0b8f78fa (127.0.98.3:43277)
I20260927 02:42:37.948854 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.945649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38623 (local address 127.0.98.2:40497)
0927 02:42:37.945711 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.945713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.945724 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:37.945757 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.945757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.945771 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.945851 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:37.946014 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.946532 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.947907 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.948480 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.948550 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.948561 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.948731 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.948779 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:42:37.948900 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.945771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36851 (local address 127.0.98.3:43277)
0927 02:42:37.945837 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.945839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.945849 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:37.945975 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.945976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.945986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.946046 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:37.946193 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.946795 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.948006 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.948583 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.948643 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.948653 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.948797 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.948833 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:42:37.948905 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.945481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40497 (local address 127.0.98.1:38623)
0927 02:42:37.945587 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.945599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.945653 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.945882 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.945883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.945886 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:37.945939 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.945941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.946608 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.946608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.947791 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.947794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.948584 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.948586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.948658 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.948664 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.948724 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.948774 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
I20260927 02:42:37.948913 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.945693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43277 (local address 127.0.98.1:36851)
0927 02:42:37.945801 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.945812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.945862 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.946058 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.946059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.946061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:37.946127 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.946129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.946884 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.946885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.947936 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.947938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.948664 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.948666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.948731 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.948736 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.948782 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.948826 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":39,"threads_started":1}
I20260927 02:42:37.949456 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e456dcbc59453c94bdd51d16398bca" candidate_uuid: "5db7bf4890bd47d38125dd42e242de88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d363b47e300a45abaab6399f1182ea65" is_pre_election: true
I20260927 02:42:37.949512 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e456dcbc59453c94bdd51d16398bca" candidate_uuid: "5db7bf4890bd47d38125dd42e242de88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" is_pre_election: true
I20260927 02:42:37.949652 7547 raft_consensus.cc:2454] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5db7bf4890bd47d38125dd42e242de88 in term 0.
I20260927 02:42:37.949685 7683 raft_consensus.cc:2454] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5db7bf4890bd47d38125dd42e242de88 in term 0.
I20260927 02:42:37.949860 7345 leader_election.cc:302] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db7bf4890bd47d38125dd42e242de88, d363b47e300a45abaab6399f1182ea65; no voters:
I20260927 02:42:37.949990 7768 raft_consensus.cc:2781] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:37.950048 7768 raft_consensus.cc:482] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:37.950071 7768 raft_consensus.cc:3037] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:37.994326 7768 raft_consensus.cc:504] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } }
I20260927 02:42:37.994558 7768 leader_election.cc:288] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [CANDIDATE]: Term 1 election: Requested vote from peers d363b47e300a45abaab6399f1182ea65 (127.0.98.2:40497), bd77f6002a0941d7acd2243f0b8f78fa (127.0.98.3:43277)
I20260927 02:42:37.994925 7547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e456dcbc59453c94bdd51d16398bca" candidate_uuid: "5db7bf4890bd47d38125dd42e242de88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d363b47e300a45abaab6399f1182ea65"
I20260927 02:42:37.994952 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e456dcbc59453c94bdd51d16398bca" candidate_uuid: "5db7bf4890bd47d38125dd42e242de88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd77f6002a0941d7acd2243f0b8f78fa"
I20260927 02:42:37.995043 7547 raft_consensus.cc:3037] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:37.995069 7683 raft_consensus.cc:3037] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:37.995899 7683 raft_consensus.cc:2454] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5db7bf4890bd47d38125dd42e242de88 in term 1.
I20260927 02:42:37.995980 7547 raft_consensus.cc:2454] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5db7bf4890bd47d38125dd42e242de88 in term 1.
I20260927 02:42:37.996143 7345 leader_election.cc:302] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db7bf4890bd47d38125dd42e242de88, bd77f6002a0941d7acd2243f0b8f78fa; no voters:
I20260927 02:42:37.996260 7768 raft_consensus.cc:2781] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:37.996423 7768 raft_consensus.cc:686] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 1 LEADER]: Becoming Leader. State: Replica: 5db7bf4890bd47d38125dd42e242de88, State: Running, Role: LEADER
I20260927 02:42:37.996537 7768 consensus_queue.cc:237] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } }
I20260927 02:42:37.997430 6493 catalog_manager.cc:5495] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5db7bf4890bd47d38125dd42e242de88 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5db7bf4890bd47d38125dd42e242de88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } health_report { overall_health: HEALTHY } } }
I20260927 02:42:38.080214 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.076994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.0.98.1:44895)
0927 02:42:38.077092 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.077094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.077104 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:38.077181 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.077182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.077215 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.077320 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:38.077482 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.078286 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.079262 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.079510 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.079646 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.079849 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:38.079964 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:38.080038 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.080074 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.080094 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":50,"threads_started":1}
I20260927 02:42:38.082172 7547 raft_consensus.cc:1261] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [term 1 FOLLOWER]: Refusing update from remote peer 5db7bf4890bd47d38125dd42e242de88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:38.082535 7773 consensus_queue.cc:1035] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:38.082651 7683 raft_consensus.cc:1261] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [term 1 FOLLOWER]: Refusing update from remote peer 5db7bf4890bd47d38125dd42e242de88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:38.082926 7773 consensus_queue.cc:1035] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:38.085479 7781 mvcc.cc:205] Tried to move back new op lower bound from 7333793620301307904 to 7333793619954409472. Current Snapshot: MvccSnapshot[applied={T|T < 7333793620301307904}]
I20260927 02:42:38.085711 7784 mvcc.cc:205] Tried to move back new op lower bound from 7333793620301307904 to 7333793619954409472. Current Snapshot: MvccSnapshot[applied={T|T < 7333793620301307904}]
I20260927 02:42:38.085906 7786 mvcc.cc:205] Tried to move back new op lower bound from 7333793620301307904 to 7333793619954409472. Current Snapshot: MvccSnapshot[applied={T|T < 7333793620301307904}]
I20260927 02:42:38.088267 6493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:38.088371 6493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:38.089956 6493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6592263c8e1d49dd9be3ddc85521a8c0: Sending DeleteTablet for 3 replicas of tablet 98e456dcbc59453c94bdd51d16398bca
I20260927 02:42:38.090260 7527 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e456dcbc59453c94bdd51d16398bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38834
I20260927 02:42:38.090332 7663 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e456dcbc59453c94bdd51d16398bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47876
I20260927 02:42:38.090482 7391 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e456dcbc59453c94bdd51d16398bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50768
I20260927 02:42:38.090690 7792 tablet_replica.cc:330] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: stopping tablet replica
I20260927 02:42:38.090807 7792 raft_consensus.cc:2229] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:38.090929 7792 raft_consensus.cc:2258] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:38.091089 7793 tablet_replica.cc:330] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: stopping tablet replica
I20260927 02:42:38.091180 7793 raft_consensus.cc:2229] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:38.091300 7793 raft_consensus.cc:2258] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:38.091296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 6461
I20260927 02:42:38.092127 7793 ts_tablet_manager.cc:1905] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:38.092499 7794 tablet_replica.cc:330] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: stopping tablet replica
I20260927 02:42:38.092669 7794 raft_consensus.cc:2229] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:38.092849 7794 raft_consensus.cc:2258] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:38.093252 7792 ts_tablet_manager.cc:1905] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:38.093251 7794 ts_tablet_manager.cc:1905] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:38.107352 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.099092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.0.98.61:33363)
0927 02:42:38.099335 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.099338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.099470 (+ 132us) server_negotiation.cc:407] Connection header received
0927 02:42:38.099499 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.099500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.099514 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.099593 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:38.100214 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.101072 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.101850 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.102190 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.102260 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.103509 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:38.103510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:38.103510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:38.103519 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:38.105298 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.105699 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.105700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.105701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.105760 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.105946 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.105946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.105947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.106622 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:38.106731 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.107152 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.107243 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":58,"threads_started":1}
I20260927 02:42:38.107373 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.099067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.0.98.60:39581)
0927 02:42:38.100513 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.100515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.100526 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:38.100548 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.100549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.100559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.100635 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:38.101860 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.102406 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.103469 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.103708 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.103767 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.104972 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:38.104973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:38.104974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:38.104981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:38.106280 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.106590 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.106591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.106592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.106643 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.106813 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.106814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.106814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.106920 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:38.107002 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.107105 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.107187 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421,"thread_start_us":69,"threads_started":1}
I20260927 02:42:38.178161 7792 ts_tablet_manager.cc:1918] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:38.178195 7793 ts_tablet_manager.cc:1918] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:38.178299 7792 log.cc:1196] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/98e456dcbc59453c94bdd51d16398bca
I20260927 02:42:38.178349 7793 log.cc:1196] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/98e456dcbc59453c94bdd51d16398bca
I20260927 02:42:38.178632 7793 ts_tablet_manager.cc:1939] T 98e456dcbc59453c94bdd51d16398bca P bd77f6002a0941d7acd2243f0b8f78fa: Deleting consensus metadata
I20260927 02:42:38.178637 7792 ts_tablet_manager.cc:1939] T 98e456dcbc59453c94bdd51d16398bca P d363b47e300a45abaab6399f1182ea65: Deleting consensus metadata
I20260927 02:42:38.204851 7794 ts_tablet_manager.cc:1918] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:38.204955 7794 log.cc:1196] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/98e456dcbc59453c94bdd51d16398bca
W20260927 02:42:38.205052 7480 connection.cc:441] server connection from 127.0.0.1:38834 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38834 (request call id 1) could send its response
I20260927 02:42:38.205228 7794 ts_tablet_manager.cc:1939] T 98e456dcbc59453c94bdd51d16398bca P 5db7bf4890bd47d38125dd42e242de88: Deleting consensus metadata
W20260927 02:42:38.205222 7618 connection.cc:441] server connection from 127.0.0.1:47876 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47876 (request call id 1) could send its response
W20260927 02:42:38.238360 7345 connection.cc:441] server connection from 127.0.0.1:50768 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50768 (request call id 1) could send its response
I20260927 02:42:39.086758 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.086457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.1:34205)
0927 02:42:39.086607 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.086700 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260927 02:42:39.086849 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.086645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:43959)
0927 02:42:39.086737 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.086800 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":54,"threads_started":1}
W20260927 02:42:39.087088 7458 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43607 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
W20260927 02:42:39.087349 7594 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43607 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
I20260927 02:42:39.088649 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.088435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.3:33187)
0927 02:42:39.088517 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.088613 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":44,"threads_started":1}
W20260927 02:42:39.088851 7730 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43607 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
I20260927 02:42:39.590376 7807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6592263c8e1d49dd9be3ddc85521a8c0)
I20260927 02:42:39.590500 7807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } }
I20260927 02:42:39.590706 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6592263c8e1d49dd9be3ddc85521a8c0)
I20260927 02:42:39.590776 7807 leader_election.cc:288] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607), 9c329716d8dd4447b1c8b9206a987619 (127.0.98.61:33363)
I20260927 02:42:39.590831 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } }
I20260927 02:42:39.591070 7808 leader_election.cc:288] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607), 821b164bce744423bda40a77e03c255a (127.0.98.60:39581)
I20260927 02:42:39.591136 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.590925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46590)
0927 02:42:39.591048 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.591098 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":49,"threads_started":1}
W20260927 02:42:39.591233 7262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
I20260927 02:42:39.591573 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.591259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46602)
0927 02:42:39.591481 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.591535 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":40,"threads_started":1}
I20260927 02:42:39.591657 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.591580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46606)
0927 02:42:39.591610 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.591632 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260927 02:42:39.591707 7077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
W20260927 02:42:39.591725 7262 leader_election.cc:334] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
I20260927 02:42:39.592134 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.592032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46614)
0927 02:42:39.592079 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.592107 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 02:42:39.592258 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
I20260927 02:42:39.594277 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.591117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.0.98.61:33363)
0927 02:42:39.591445 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.591447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.591454 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:42:39.591480 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.591481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.591492 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.591933 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:39.591968 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.592649 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.593609 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.593979 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.594036 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.594047 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.594152 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.594188 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":58,"threads_started":1}
I20260927 02:42:39.594278 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.590951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33363 (local address 127.0.0.1:52696)
0927 02:42:39.591342 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.591352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.591405 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.591590 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.591591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.591593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:39.591856 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.591859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.592725 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.592725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.593532 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.593534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.594043 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.594044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.594095 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.594099 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.594141 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.594180 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":47,"threads_started":1}
I20260927 02:42:39.594293 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.591190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39581 (local address 127.0.0.1:60108)
0927 02:42:39.591423 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.591433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.591495 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.591716 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.591717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.591720 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:39.591774 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.591776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.592668 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.592669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.593540 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.593542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.594058 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.594059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.594115 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.594118 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.594160 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.594206 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260927 02:42:39.594333 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.591311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.0.98.60:39581)
0927 02:42:39.591442 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.591444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.591551 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:42:39.591616 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.591617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.591626 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.591682 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:39.591855 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.592581 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.593620 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.593992 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.594052 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.594062 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.594174 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.594210 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260927 02:42:39.594660 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c329716d8dd4447b1c8b9206a987619" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "821b164bce744423bda40a77e03c255a" is_pre_election: true
I20260927 02:42:39.594661 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "821b164bce744423bda40a77e03c255a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c329716d8dd4447b1c8b9206a987619" is_pre_election: true
I20260927 02:42:39.594779 7286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c329716d8dd4447b1c8b9206a987619 in term 1.
I20260927 02:42:39.594780 7101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821b164bce744423bda40a77e03c255a in term 1.
I20260927 02:42:39.594926 7076 leader_election.cc:302] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 821b164bce744423bda40a77e03c255a, 9c329716d8dd4447b1c8b9206a987619; no voters: 6592263c8e1d49dd9be3ddc85521a8c0
I20260927 02:42:39.594940 7263 leader_election.cc:302] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 821b164bce744423bda40a77e03c255a, 9c329716d8dd4447b1c8b9206a987619; no voters: 6592263c8e1d49dd9be3ddc85521a8c0
I20260927 02:42:39.595042 7807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:42:39.595034 7808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:42:39.595077 7807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6592263c8e1d49dd9be3ddc85521a8c0)
I20260927 02:42:39.595077 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6592263c8e1d49dd9be3ddc85521a8c0)
I20260927 02:42:39.595090 7807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:42:39.595098 7808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:42:39.595739 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } }
I20260927 02:42:39.595750 7807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } }
I20260927 02:42:39.595857 7807 leader_election.cc:288] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [CANDIDATE]: Term 2 election: Requested vote from peers 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607), 9c329716d8dd4447b1c8b9206a987619 (127.0.98.61:33363)
I20260927 02:42:39.595866 7808 leader_election.cc:288] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 2 election: Requested vote from peers 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607), 821b164bce744423bda40a77e03c255a (127.0.98.60:39581)
I20260927 02:42:39.595988 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c329716d8dd4447b1c8b9206a987619" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "821b164bce744423bda40a77e03c255a"
I20260927 02:42:39.596026 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.595947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46630)
0927 02:42:39.595976 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.596001 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 02:42:39.596081 7286 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c329716d8dd4447b1c8b9206a987619 in current term 2: Already voted for candidate 821b164bce744423bda40a77e03c255a in this term.
I20260927 02:42:39.596145 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "821b164bce744423bda40a77e03c255a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c329716d8dd4447b1c8b9206a987619"
I20260927 02:42:39.596190 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.596038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46644)
0927 02:42:39.596110 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.596139 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 02:42:39.596248 7101 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 821b164bce744423bda40a77e03c255a in current term 2: Already voted for candidate 9c329716d8dd4447b1c8b9206a987619 in this term.
I20260927 02:42:39.596504 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.596421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46652)
0927 02:42:39.596453 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.596481 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 02:42:39.596570 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
I20260927 02:42:39.596619 7077 leader_election.cc:302] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c329716d8dd4447b1c8b9206a987619; no voters: 6592263c8e1d49dd9be3ddc85521a8c0, 821b164bce744423bda40a77e03c255a
I20260927 02:42:39.596694 7808 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 02:42:39.596779 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.596531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:46664)
0927 02:42:39.596730 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.596758 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
W20260927 02:42:39.596841 7262 leader_election.cc:334] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
I20260927 02:42:39.596877 7262 leader_election.cc:302] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 821b164bce744423bda40a77e03c255a; no voters: 6592263c8e1d49dd9be3ddc85521a8c0, 9c329716d8dd4447b1c8b9206a987619
I20260927 02:42:39.596935 7807 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 02:42:40.088109 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:40.087531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.1:47393)
0927 02:42:40.088002 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:40.088051 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":127,"threads_started":1}
I20260927 02:42:40.088356 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:40.087702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:51011)
0927 02:42:40.088143 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:40.088309 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":42,"threads_started":1}
I20260927 02:42:40.089316 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:40.089129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.3:44223)
0927 02:42:40.089250 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:40.089282 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260927 02:42:41.089073 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.088682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.1:37281)
0927 02:42:41.088946 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.089004 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":64,"threads_started":1}
I20260927 02:42:41.089149 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.088986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:40483)
0927 02:42:41.089079 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.089111 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
I20260927 02:42:41.089859 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.089695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.3:46507)
0927 02:42:41.089795 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.089823 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":45,"threads_started":1}
I20260927 02:42:41.346990 7823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:41.347081 7823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } }
I20260927 02:42:41.347251 7823 leader_election.cc:288] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607), 821b164bce744423bda40a77e03c255a (127.0.98.60:39581)
I20260927 02:42:41.347457 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c329716d8dd4447b1c8b9206a987619" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "821b164bce744423bda40a77e03c255a" is_pre_election: true
I20260927 02:42:41.347536 7286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c329716d8dd4447b1c8b9206a987619 in term 2.
I20260927 02:42:41.347532 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.347364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:55212)
0927 02:42:41.347456 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.347492 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":44,"threads_started":1}
I20260927 02:42:41.347669 7076 leader_election.cc:302] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 821b164bce744423bda40a77e03c255a, 9c329716d8dd4447b1c8b9206a987619; no voters:
I20260927 02:42:41.347757 7823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 02:42:41.347792 7823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:41.347811 7823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:42:41.347978 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.347892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:55228)
0927 02:42:41.347926 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.347956 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 02:42:41.348053 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
I20260927 02:42:41.348596 7823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } }
I20260927 02:42:41.348708 7823 leader_election.cc:288] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 3 election: Requested vote from peers 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607), 821b164bce744423bda40a77e03c255a (127.0.98.60:39581)
I20260927 02:42:41.348884 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.348789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:55240)
0927 02:42:41.348824 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.348859 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:42:41.348935 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c329716d8dd4447b1c8b9206a987619" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "821b164bce744423bda40a77e03c255a"
I20260927 02:42:41.349014 7286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:42:41.349256 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.349157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:55246)
0927 02:42:41.349206 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.349239 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:42:41.349325 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
I20260927 02:42:41.349745 7286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c329716d8dd4447b1c8b9206a987619 in term 3.
I20260927 02:42:41.349905 7076 leader_election.cc:302] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 821b164bce744423bda40a77e03c255a, 9c329716d8dd4447b1c8b9206a987619; no voters: 6592263c8e1d49dd9be3ddc85521a8c0
I20260927 02:42:41.349977 7823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:42:41.350107 7823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [term 3 LEADER]: Becoming Leader. State: Replica: 9c329716d8dd4447b1c8b9206a987619, State: Running, Role: LEADER
I20260927 02:42:41.350183 7823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } }
I20260927 02:42:41.350502 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c329716d8dd4447b1c8b9206a987619. Latest consensus state: current_term: 3 leader_uuid: "9c329716d8dd4447b1c8b9206a987619" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:41.350627 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:41.350807 7828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:41.351300 7828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:41.351372 7828 catalog_manager.cc:1259] Loaded cluster ID: e2e9a06fcbd5484fa0b2e57309ce8ba0
I20260927 02:42:41.351394 7828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:41.351553 7828 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:41.351608 7828 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Loaded TSK: 0
I20260927 02:42:41.351702 7828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:41.433781 7091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:41.435480 7091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:41.436626 7286 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [term 3 FOLLOWER]: Refusing update from remote peer 9c329716d8dd4447b1c8b9206a987619: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 02:42:41.436625 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.436472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:55274)
0927 02:42:41.436547 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.436588 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 02:42:41.436924 7826 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 02:42:41.437034 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.436937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.0.1:55276)
0927 02:42:41.436984 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.437013 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:42:41.437088 7077 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 -> Peer 6592263c8e1d49dd9be3ddc85521a8c0 (127.0.98.62:43607): Couldn't send request to peer 6592263c8e1d49dd9be3ddc85521a8c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:42:41.437578 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c329716d8dd4447b1c8b9206a987619. Latest consensus state: current_term: 3 leader_uuid: "9c329716d8dd4447b1c8b9206a987619" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:41.437693 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:42:41.438071 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9c329716d8dd4447b1c8b9206a987619" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:41.438167 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:41.438310 7831 mvcc.cc:205] Tried to move back new op lower bound from 7333793634042023936 to 7333793633691013120. Current Snapshot: MvccSnapshot[applied={T|T < 7333793634042023936}]
I20260927 02:42:41.438467 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9c329716d8dd4447b1c8b9206a987619" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6592263c8e1d49dd9be3ddc85521a8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43607 } } peers { permanent_uuid: "9c329716d8dd4447b1c8b9206a987619" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33363 } attrs { promote: false } } peers { permanent_uuid: "821b164bce744423bda40a77e03c255a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39581 } attrs { promote: false } } }
I20260927 02:42:41.438555 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821b164bce744423bda40a77e03c255a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:42:41.443171 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.439902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44895 (local address 127.0.0.1:58316)
0927 02:42:41.439941 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.439950 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.439993 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.440648 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.440649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.440651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:41.440684 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.440685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.441404 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.441405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.442322 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.442324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.442937 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.442938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.442995 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.443000 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.443044 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.443085 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:42:41.443174 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.439969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.0.98.1:44895)
0927 02:42:41.440253 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.440256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.440262 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:42:41.440581 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.440582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.440592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.440641 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:41.440757 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.441321 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.442412 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.442841 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.442935 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.442947 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.443059 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.443098 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":78,"threads_started":1}
I20260927 02:42:41.443426 7391 tablet_service.cc:1467] Processing CreateTablet for tablet f122b808e6904886ad8a7d3e5ef5e0c8 (DEFAULT_TABLE table=test-table [id=6fcfef2eadbf4497908697c1687a9d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:41.443562 7391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f122b808e6904886ad8a7d3e5ef5e0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:41.443779 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.440135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43277 (local address 127.0.0.1:42666)
0927 02:42:41.440458 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.440467 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.440512 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.440805 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.440806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.440808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:41.440849 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.440850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.441722 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.441723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.442819 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.442820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.443607 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.443607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.443654 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.443657 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.443696 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.443723 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":59,"threads_started":1}
I20260927 02:42:41.443810 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.440263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.98.3:43277)
0927 02:42:41.440433 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.440434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.440672 (+ 238us) server_negotiation.cc:407] Connection header received
0927 02:42:41.440695 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.440696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.440704 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.440764 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:41.440922 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.441624 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.442896 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.443528 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.443606 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.443617 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.443710 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.443750 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
I20260927 02:42:41.444682 7663 tablet_service.cc:1467] Processing CreateTablet for tablet f122b808e6904886ad8a7d3e5ef5e0c8 (DEFAULT_TABLE table=test-table [id=6fcfef2eadbf4497908697c1687a9d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:41.444804 7663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f122b808e6904886ad8a7d3e5ef5e0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:41.444939 7840 tablet_bootstrap.cc:492] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88: Bootstrap starting.
I20260927 02:42:41.445098 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.439971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40497 (local address 127.0.0.1:51016)
0927 02:42:41.440100 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.440109 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.440156 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.440408 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.440410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.440412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:41.440469 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.440471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.441469 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.441470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.442655 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.442657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.444856 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.444857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.444922 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.444927 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.444976 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.445017 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":34,"thread_start_us":65,"threads_started":1}
I20260927 02:42:41.445253 7840 tablet_bootstrap.cc:654] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:41.445297 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.440038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.0.98.2:40497)
0927 02:42:41.440250 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.440252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.440261 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:41.440286 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.440286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.440297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.440378 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:41.440575 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.441330 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.442742 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.443273 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.443342 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.443355 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.445179 (+ 1824us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.445231 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":48,"threads_started":1}
I20260927 02:42:41.445501 7527 tablet_service.cc:1467] Processing CreateTablet for tablet f122b808e6904886ad8a7d3e5ef5e0c8 (DEFAULT_TABLE table=test-table [id=6fcfef2eadbf4497908697c1687a9d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:41.445612 7527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f122b808e6904886ad8a7d3e5ef5e0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:41.445736 7840 tablet_bootstrap.cc:492] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88: No bootstrap required, opened a new log
I20260927 02:42:41.445770 7840 ts_tablet_manager.cc:1397] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:41.445851 7840 raft_consensus.cc:348] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } }
I20260927 02:42:41.445899 7840 raft_consensus.cc:374] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:41.445911 7840 raft_consensus.cc:729] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db7bf4890bd47d38125dd42e242de88, State: Initialized, Role: FOLLOWER
I20260927 02:42:41.445941 7840 consensus_queue.cc:260] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } }
I20260927 02:42:41.446069 7840 ts_tablet_manager.cc:1428] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.446341 7843 tablet_bootstrap.cc:492] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: Bootstrap starting.
I20260927 02:42:41.446588 7843 tablet_bootstrap.cc:654] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:41.446985 7845 tablet_bootstrap.cc:492] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: Bootstrap starting.
I20260927 02:42:41.447242 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.447113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.1:34923)
0927 02:42:41.447171 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.447214 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:42:41.447379 7845 tablet_bootstrap.cc:654] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:41.447767 7843 tablet_bootstrap.cc:492] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: No bootstrap required, opened a new log
I20260927 02:42:41.447857 7843 ts_tablet_manager.cc:1397] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:41.447976 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.447866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.1:45857)
0927 02:42:41.447910 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.447953 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 02:42:41.448064 7459 heartbeater.cc:503] Master 127.0.98.61:33363 was elected leader, sending a full tablet report...
I20260927 02:42:41.448043 7843 raft_consensus.cc:348] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } }
I20260927 02:42:41.448123 7843 raft_consensus.cc:374] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:41.448151 7843 raft_consensus.cc:729] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd77f6002a0941d7acd2243f0b8f78fa, State: Initialized, Role: FOLLOWER
I20260927 02:42:41.448215 7843 consensus_queue.cc:260] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } }
I20260927 02:42:41.448426 7843 ts_tablet_manager.cc:1428] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:42:41.448673 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.448551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.3:37193)
0927 02:42:41.448614 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.448645 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 02:42:41.448803 7845 tablet_bootstrap.cc:492] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: No bootstrap required, opened a new log
I20260927 02:42:41.448853 7845 ts_tablet_manager.cc:1397] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:41.448980 7845 raft_consensus.cc:348] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } }
I20260927 02:42:41.449005 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.448877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.3:48375)
0927 02:42:41.448959 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.448982 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 02:42:41.449150 7845 raft_consensus.cc:374] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:41.449218 7845 raft_consensus.cc:729] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d363b47e300a45abaab6399f1182ea65, State: Initialized, Role: FOLLOWER
I20260927 02:42:41.449244 7731 heartbeater.cc:503] Master 127.0.98.61:33363 was elected leader, sending a full tablet report...
I20260927 02:42:41.449278 7845 consensus_queue.cc:260] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } }
I20260927 02:42:41.449455 7845 ts_tablet_manager.cc:1428] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:42:41.449736 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.449652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:34405)
0927 02:42:41.449686 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.449714 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 02:42:41.449865 7595 heartbeater.cc:503] Master 127.0.98.61:33363 was elected leader, sending a full tablet report...
I20260927 02:42:41.450060 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.449982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:46085)
0927 02:42:41.450014 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.450040 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:42:41.599315 7848 raft_consensus.cc:482] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:41.599432 7848 raft_consensus.cc:504] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } }
I20260927 02:42:41.599810 7848 leader_election.cc:288] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5db7bf4890bd47d38125dd42e242de88 (127.0.98.1:44895), bd77f6002a0941d7acd2243f0b8f78fa (127.0.98.3:43277)
I20260927 02:42:41.602721 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.600112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43277 (local address 127.0.98.2:50827)
0927 02:42:41.600199 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.600212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.600285 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.600444 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.600445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.600447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:41.600521 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.600522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.601186 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.601187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.601962 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.601963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.602476 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.602477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.602543 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.602546 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.602592 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.602630 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 02:42:41.602705 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.600226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50827 (local address 127.0.98.3:43277)
0927 02:42:41.600272 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.600274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.600285 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:41.600346 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.600347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.600357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.600437 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:41.600623 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.601108 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.602049 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.602402 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.602462 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.602474 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.602606 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.602638 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:42:41.602962 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f122b808e6904886ad8a7d3e5ef5e0c8" candidate_uuid: "d363b47e300a45abaab6399f1182ea65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" is_pre_election: true
I20260927 02:42:41.603034 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.600118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44895 (local address 127.0.98.2:51289)
0927 02:42:41.600374 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.600409 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.600478 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.600651 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.600651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.600653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:41.600716 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.600717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.601338 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.601339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.602203 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.602206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.602772 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.602773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.602844 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.602847 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.602910 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.602946 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":83,"thread_start_us":104,"threads_started":1}
I20260927 02:42:41.603040 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.600220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51289 (local address 127.0.98.1:44895)
0927 02:42:41.600296 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.600298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.600464 (+ 166us) server_negotiation.cc:407] Connection header received
0927 02:42:41.600556 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.600557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.600571 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.600624 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:41.600777 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.601244 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.602304 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.602674 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.602750 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.602762 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.602936 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.602982 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:42:41.603065 7683 raft_consensus.cc:2454] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d363b47e300a45abaab6399f1182ea65 in term 0.
I20260927 02:42:41.603255 7481 leader_election.cc:302] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd77f6002a0941d7acd2243f0b8f78fa, d363b47e300a45abaab6399f1182ea65; no voters:
I20260927 02:42:41.603333 7411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f122b808e6904886ad8a7d3e5ef5e0c8" candidate_uuid: "d363b47e300a45abaab6399f1182ea65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db7bf4890bd47d38125dd42e242de88" is_pre_election: true
I20260927 02:42:41.603425 7848 raft_consensus.cc:2781] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:41.603457 7411 raft_consensus.cc:2454] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d363b47e300a45abaab6399f1182ea65 in term 0.
I20260927 02:42:41.603475 7848 raft_consensus.cc:482] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:41.603498 7848 raft_consensus.cc:3037] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:41.604261 7848 raft_consensus.cc:504] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } }
I20260927 02:42:41.604424 7848 leader_election.cc:288] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [CANDIDATE]: Term 1 election: Requested vote from peers 5db7bf4890bd47d38125dd42e242de88 (127.0.98.1:44895), bd77f6002a0941d7acd2243f0b8f78fa (127.0.98.3:43277)
I20260927 02:42:41.604571 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f122b808e6904886ad8a7d3e5ef5e0c8" candidate_uuid: "d363b47e300a45abaab6399f1182ea65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd77f6002a0941d7acd2243f0b8f78fa"
I20260927 02:42:41.604601 7411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f122b808e6904886ad8a7d3e5ef5e0c8" candidate_uuid: "d363b47e300a45abaab6399f1182ea65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db7bf4890bd47d38125dd42e242de88"
I20260927 02:42:41.604647 7683 raft_consensus.cc:3037] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:41.604661 7411 raft_consensus.cc:3037] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:41.605391 7411 raft_consensus.cc:2454] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d363b47e300a45abaab6399f1182ea65 in term 1.
I20260927 02:42:41.605391 7683 raft_consensus.cc:2454] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d363b47e300a45abaab6399f1182ea65 in term 1.
I20260927 02:42:41.605557 7481 leader_election.cc:302] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd77f6002a0941d7acd2243f0b8f78fa, d363b47e300a45abaab6399f1182ea65; no voters:
I20260927 02:42:41.605676 7848 raft_consensus.cc:2781] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:41.605841 7848 raft_consensus.cc:686] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 1 LEADER]: Becoming Leader. State: Replica: d363b47e300a45abaab6399f1182ea65, State: Running, Role: LEADER
I20260927 02:42:41.605916 7848 consensus_queue.cc:237] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } }
I20260927 02:42:41.606137 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.606029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:39715)
0927 02:42:41.606080 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.606110 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:42:41.606624 7091 catalog_manager.cc:5495] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 reported cstate change: term changed from 0 to 1, leader changed from <none> to d363b47e300a45abaab6399f1182ea65 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d363b47e300a45abaab6399f1182ea65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d363b47e300a45abaab6399f1182ea65" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:41.625869 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.623282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.0.98.2:40497)
0927 02:42:41.623336 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.623338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.623345 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:42:41.623382 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.623382 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.623400 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.623463 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:41.623612 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.624136 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.625009 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.625244 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.625307 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.625523 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:41.625628 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:41.625693 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.625765 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.625786 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:42:41.627071 7683 raft_consensus.cc:1261] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [term 1 FOLLOWER]: Refusing update from remote peer d363b47e300a45abaab6399f1182ea65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:41.627139 7411 raft_consensus.cc:1261] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [term 1 FOLLOWER]: Refusing update from remote peer d363b47e300a45abaab6399f1182ea65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:41.627460 7848 consensus_queue.cc:1035] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db7bf4890bd47d38125dd42e242de88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:41.627604 7850 consensus_queue.cc:1035] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd77f6002a0941d7acd2243f0b8f78fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:41.628502 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.628328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.1:60605)
0927 02:42:41.628425 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.628467 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 02:42:41.628773 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.628679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.3:56495)
0927 02:42:41.628715 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.628743 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 02:42:41.630244 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.629888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:36033)
0927 02:42:41.630180 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.630213 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274}
I20260927 02:42:41.630611 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.630514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.1:46515)
0927 02:42:41.630552 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.630576 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 02:42:41.631119 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.631032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.3:52343)
0927 02:42:41.631065 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.631090 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:42:41.631820 7089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:41.631908 7089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:41.631916 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.631697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43607 (local address 127.0.98.2:58993)
0927 02:42:41.631855 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.631887 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260927 02:42:41.633147 7089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c329716d8dd4447b1c8b9206a987619: Sending DeleteTablet for 3 replicas of tablet f122b808e6904886ad8a7d3e5ef5e0c8
I20260927 02:42:41.633419 7527 tablet_service.cc:1514] Processing DeleteTablet for tablet f122b808e6904886ad8a7d3e5ef5e0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:41 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:51016
I20260927 02:42:41.633420 7391 tablet_service.cc:1514] Processing DeleteTablet for tablet f122b808e6904886ad8a7d3e5ef5e0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:41 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58316
I20260927 02:42:41.633589 7663 tablet_service.cc:1514] Processing DeleteTablet for tablet f122b808e6904886ad8a7d3e5ef5e0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:41 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42666
I20260927 02:42:41.633611 7863 tablet_replica.cc:330] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88: stopping tablet replica
I20260927 02:42:41.633666 7863 raft_consensus.cc:2229] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:41.633706 7863 raft_consensus.cc:2258] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:41.633831 7864 tablet_replica.cc:330] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: stopping tablet replica
I20260927 02:42:41.633960 7865 tablet_replica.cc:330] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: stopping tablet replica
I20260927 02:42:41.634051 7864 raft_consensus.cc:2229] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:41.634110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 7328
I20260927 02:42:41.634132 7865 raft_consensus.cc:2229] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:41.634135 7863 ts_tablet_manager.cc:1905] T f122b808e6904886ad8a7d3e5ef5e0c8 P 5db7bf4890bd47d38125dd42e242de88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:41.634177 7865 raft_consensus.cc:2258] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:41.634279 7864 raft_consensus.cc:2258] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:41.635066 7865 ts_tablet_manager.cc:1905] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:41.637051 7865 ts_tablet_manager.cc:1918] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:41.637125 7865 log.cc:1196] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/f122b808e6904886ad8a7d3e5ef5e0c8
I20260927 02:42:41.637490 7865 ts_tablet_manager.cc:1939] T f122b808e6904886ad8a7d3e5ef5e0c8 P bd77f6002a0941d7acd2243f0b8f78fa: Deleting consensus metadata
I20260927 02:42:41.637691 7864 ts_tablet_manager.cc:1905] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:41.639611 7864 ts_tablet_manager.cc:1918] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:41.639753 7864 log.cc:1196] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/f122b808e6904886ad8a7d3e5ef5e0c8
I20260927 02:42:41.639765 7076 catalog_manager.cc:4841] TS bd77f6002a0941d7acd2243f0b8f78fa (127.0.98.3:43277): tablet f122b808e6904886ad8a7d3e5ef5e0c8 (table test-table [id=6fcfef2eadbf4497908697c1687a9d97]) successfully deleted
I20260927 02:42:41.640014 7864 ts_tablet_manager.cc:1939] T f122b808e6904886ad8a7d3e5ef5e0c8 P d363b47e300a45abaab6399f1182ea65: Deleting consensus metadata
I20260927 02:42:41.640429 7076 catalog_manager.cc:4841] TS d363b47e300a45abaab6399f1182ea65 (127.0.98.2:40497): tablet f122b808e6904886ad8a7d3e5ef5e0c8 (table test-table [id=6fcfef2eadbf4497908697c1687a9d97]) successfully deleted
W20260927 02:42:41.641211 7835 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:41.640803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44895 (local address 127.0.0.1:58320)
0927 02:42:41.640843 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.640853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.640891 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.641164 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44895: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":19}
W20260927 02:42:41.641343 7078 catalog_manager.cc:4584] TS 5db7bf4890bd47d38125dd42e242de88 (127.0.98.1:44895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f122b808e6904886ad8a7d3e5ef5e0c8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44895: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:42:41.641556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 7464
I20260927 02:42:41.646612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 7600
I20260927 02:42:41.650738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 6957
I20260927 02:42:41.654691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 7145
2026-09-27T02:42:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6974 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:41 dist-test-slave-44kr krb5kdc[7868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:41 dist-test-slave-44kr krb5kdc[7868](info): set up 2 sockets
Sep 27 02:42:41 dist-test-slave-44kr krb5kdc[7868](info): commencing operation
krb5kdc: starting...
W20260927 02:42:42.616681 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:42 dist-test-slave-44kr krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:42:42.739228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:42.821033 7881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:42.821256 7881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:42.821280 7881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:42.822427 7881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:42.822463 7881 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:42:42.822475 7881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:42.822486 7881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:42.822499 7881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:42.822508 7881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:42.823956 7881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43353
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:42.824172 7881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:42.824368 7881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:42.829797 7886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:42.829762 7887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:42.829871 7881 server_base.cc:1034] running on GCE node
W20260927 02:42:42.829777 7889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:42.830260 7881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:42.830549 7881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:42.831715 7881 hybrid_clock.cc:648] HybridClock initialized: now 1790476962831697 us; error 31 us; skew 500 ppm
Sep 27 02:42:42 dist-test-slave-44kr krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:42.834651 7881 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:42:42.835153 7881 webserver.cc:460] Webserver started at http://127.0.98.62:40031/ using document root <none> and password file <none>
I20260927 02:42:42.835348 7881 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:42.835389 7881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:42.835489 7881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:42.836609 7881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "431662ff84e6402f97495d3dc2a5aedf"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-44kr"
server_key: "f980f99c4e0fc5e26bb5379fa136f921"
server_key_iv: "4022afa3bd5abc9b918e0e969f5d23a6"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.836928 7881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "431662ff84e6402f97495d3dc2a5aedf"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-44kr"
server_key: "f980f99c4e0fc5e26bb5379fa136f921"
server_key_iv: "4022afa3bd5abc9b918e0e969f5d23a6"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.838358 7881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:42.839207 7896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.839485 7881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:42.839589 7881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "431662ff84e6402f97495d3dc2a5aedf"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-44kr"
server_key: "f980f99c4e0fc5e26bb5379fa136f921"
server_key_iv: "4022afa3bd5abc9b918e0e969f5d23a6"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.839668 7881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:42.859004 7881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:42.861218 7881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:42.861354 7881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:42.865785 7881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43353
I20260927 02:42:42.865836 7958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43353 every 8 connection(s)
I20260927 02:42:42.866135 7881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:42.866752 7959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:42.869153 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: Bootstrap starting.
I20260927 02:42:42.869727 7959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:42.869983 7959 log.cc:824] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:42.870599 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: No bootstrap required, opened a new log
I20260927 02:42:42.872627 7959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431662ff84e6402f97495d3dc2a5aedf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43353 } }
I20260927 02:42:42.872774 7959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:42.872807 7959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431662ff84e6402f97495d3dc2a5aedf, State: Initialized, Role: FOLLOWER
I20260927 02:42:42.872941 7959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431662ff84e6402f97495d3dc2a5aedf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43353 } }
I20260927 02:42:42.873003 7959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:42.873031 7959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:42.873078 7959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:42.873585 7959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431662ff84e6402f97495d3dc2a5aedf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43353 } }
I20260927 02:42:42.873679 7959 leader_election.cc:302] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 431662ff84e6402f97495d3dc2a5aedf; no voters:
I20260927 02:42:42.873848 7959 leader_election.cc:288] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:42.873980 7962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:42.874101 7959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:42.874150 7962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [term 1 LEADER]: Becoming Leader. State: Replica: 431662ff84e6402f97495d3dc2a5aedf, State: Running, Role: LEADER
I20260927 02:42:42.874320 7962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431662ff84e6402f97495d3dc2a5aedf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43353 } }
I20260927 02:42:42.874419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 7881
I20260927 02:42:42.874504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:42.874703 392 external_mini_cluster.cc:1428] Setting key d3aad3b66425efc8419f1db58b1cd30b
I20260927 02:42:42.874693 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "431662ff84e6402f97495d3dc2a5aedf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431662ff84e6402f97495d3dc2a5aedf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43353 } } }
I20260927 02:42:42.874780 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:42.875017 7964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 431662ff84e6402f97495d3dc2a5aedf. Latest consensus state: current_term: 1 leader_uuid: "431662ff84e6402f97495d3dc2a5aedf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431662ff84e6402f97495d3dc2a5aedf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43353 } } }
I20260927 02:42:42.875147 7964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf [sys.catalog]: This master's current role is: LEADER
W20260927 02:42:42.875989 7979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:42.876096 7979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:42.876205 7971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:42.876480 7971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:42.878491 7971 catalog_manager.cc:1347] Generated new cluster ID: bbc441db4c814549997205e06bddc6b5
I20260927 02:42:42.878556 7971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:42:42 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:42.886096 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.875404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.0.98.62:43353)
0927 02:42:42.875557 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.875598 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.875633 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:42:42.875768 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.875775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.875822 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.875986 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:42.876812 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.877987 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.878989 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.879663 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.879790 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.882527 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:42.882532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:42.882532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:42.882547 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:42.884448 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:42.884870 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:42.884873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:42.884874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:42.884980 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:42.885171 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:42.885172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:42.885172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:42.885479 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:42.885622 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.885733 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.885869 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260927 02:42:42.903975 7971 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:42.904654 7971 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:42.914098 7971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: Generated new TSK 0
I20260927 02:42:42.914340 7971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:42:42.951817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43353
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:43.033847 7985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:43.034068 7985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:43.034096 7985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:43.035219 7985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:43.035257 7985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:43.035276 7985 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:42:43.035292 7985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:43.036839 7985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:43.037052 7985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:43.037250 7985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:43.042738 7991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:43.042781 7990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:43.042799 7985 server_base.cc:1034] running on GCE node
W20260927 02:42:43.042738 7993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:43.043160 7985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:43.043447 7985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:43.044643 7985 hybrid_clock.cc:648] HybridClock initialized: now 1790476963044609 us; error 50 us; skew 500 ppm
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476963, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:43.047535 7985 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:42:43.047997 7985 webserver.cc:460] Webserver started at http://127.0.98.1:45501/ using document root <none> and password file <none>
I20260927 02:42:43.048192 7985 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:43.048234 7985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:43.048326 7985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:43.049407 7985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "8d50c57e6ed64cd9a8c159569e28e610"
format_stamp: "Formatted at 2026-09-27 02:42:43 on dist-test-slave-44kr"
server_key: "b65dbbff3cf17fee539c4a6384d20ab1"
server_key_iv: "e5fe4c6525333804848381032489a7fb"
server_key_version: "encryptionkey@0"
I20260927 02:42:43.049719 7985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8d50c57e6ed64cd9a8c159569e28e610"
format_stamp: "Formatted at 2026-09-27 02:42:43 on dist-test-slave-44kr"
server_key: "b65dbbff3cf17fee539c4a6384d20ab1"
server_key_iv: "e5fe4c6525333804848381032489a7fb"
server_key_version: "encryptionkey@0"
I20260927 02:42:43.050971 7985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:42:43.051755 8000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:43.051985 7985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:43.052099 7985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "8d50c57e6ed64cd9a8c159569e28e610"
format_stamp: "Formatted at 2026-09-27 02:42:43 on dist-test-slave-44kr"
server_key: "b65dbbff3cf17fee539c4a6384d20ab1"
server_key_iv: "e5fe4c6525333804848381032489a7fb"
server_key_version: "encryptionkey@0"
I20260927 02:42:43.052175 7985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:43.067646 7985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:43.069952 7985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:43.070104 7985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:43.070626 7985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:43.070674 7985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:43.070706 7985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:43.070724 7985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476963, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:43.077962 7985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46107
I20260927 02:42:43.078044 8115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46107 every 8 connection(s)
I20260927 02:42:43.078346 7985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:43.081072 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.071255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49845 (local address 127.0.98.62:43353)
0927 02:42:43.071315 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.071317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.071519 (+ 202us) server_negotiation.cc:407] Connection header received
0927 02:42:43.072056 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.072058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.072071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.072137 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:43.073175 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.073790 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.075248 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.075486 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.075552 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.078306 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:43.078308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:43.078308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:43.078320 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:43.079706 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.080091 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.080093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.080093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.080153 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.080373 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.080374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.080374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.080529 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:43.080595 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.080862 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.080979 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:42:43.081241 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.071200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43353 (local address 127.0.98.1:49845)
0927 02:42:43.071391 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.071433 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.071954 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.072287 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.072293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.072713 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:43.073027 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.073035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.073891 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.073892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.075161 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.075164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.075576 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.075577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.075733 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.076494 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:43.076503 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:43.078199 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:43.079787 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.079793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.079795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.080016 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.080208 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.080208 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.080209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.080318 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.080587 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:43.080593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:43.080739 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.080831 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.080989 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260927 02:42:43.081737 8116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43353
I20260927 02:42:43.081820 8116 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:43.081990 8116 heartbeater.cc:511] Master 127.0.98.62:43353 requested a full tablet report, sending...
I20260927 02:42:43.082520 7912 ts_manager.cc:194] Registered new tserver with Master: 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1:46107)
I20260927 02:42:43.083303 7912 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49845
I20260927 02:42:43.087136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 7985
I20260927 02:42:43.087239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:43.087445 392 external_mini_cluster.cc:1428] Setting key 9c7791d516db55c479b66049aef8209b
I20260927 02:42:43.088547 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.082854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44785 (local address 127.0.98.62:43353)
0927 02:42:43.082891 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.082892 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.082980 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:42:43.083015 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.083016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.083026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.083086 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:43.083347 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.084011 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.084782 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.084966 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.085023 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.085829 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:43.085830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:43.085830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:43.085835 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:43.087498 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.087819 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.087820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.087821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.087870 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.088031 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.088032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.088032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.088183 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:43.088244 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.088408 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.088487 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:42:43.088590 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.082780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43353 (local address 127.0.98.1:44785)
0927 02:42:43.082920 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.082929 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.082980 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.083089 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.083089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.083241 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:43.083292 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.083293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.084094 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.084095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.084710 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.084713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.085025 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.085026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.085070 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.085481 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:43.085485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:43.085755 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:43.087571 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.087573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.087573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.087752 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.087923 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.087923 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.087924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.087984 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.088241 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:43.088242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:43.088289 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.088364 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.088493 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:42:43.112142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43353
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:43.194317 8125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:43.194553 8125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:43.194598 8125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:43.195942 8125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:43.196007 8125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:43.196035 8125 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:42:43.196060 8125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:43.197829 8125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:43.198130 8125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:43.198367 8125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:43.203538 8133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:43.203599 8130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:43.203538 8131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:43.203830 8125 server_base.cc:1034] running on GCE node
I20260927 02:42:43.204025 8125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:43.204303 8125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:43.205497 8125 hybrid_clock.cc:648] HybridClock initialized: now 1790476963205499 us; error 44 us; skew 500 ppm
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476963, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:43.208489 8125 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:42:43.209000 8125 webserver.cc:460] Webserver started at http://127.0.98.2:46735/ using document root <none> and password file <none>
I20260927 02:42:43.209281 8125 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:43.209333 8125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:43.209470 8125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:43.210565 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "f63b020da5194b2e95d5d8403bba5b69"
format_stamp: "Formatted at 2026-09-27 02:42:43 on dist-test-slave-44kr"
server_key: "ce154f88ff0da0bd3f5dcee95bffd380"
server_key_iv: "206455f5daeaeb6b4306f66ce067a49d"
server_key_version: "encryptionkey@0"
I20260927 02:42:43.210918 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f63b020da5194b2e95d5d8403bba5b69"
format_stamp: "Formatted at 2026-09-27 02:42:43 on dist-test-slave-44kr"
server_key: "ce154f88ff0da0bd3f5dcee95bffd380"
server_key_iv: "206455f5daeaeb6b4306f66ce067a49d"
server_key_version: "encryptionkey@0"
I20260927 02:42:43.212494 8125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:43.213380 8140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:43.213584 8125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:43.213677 8125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "f63b020da5194b2e95d5d8403bba5b69"
format_stamp: "Formatted at 2026-09-27 02:42:43 on dist-test-slave-44kr"
server_key: "ce154f88ff0da0bd3f5dcee95bffd380"
server_key_iv: "206455f5daeaeb6b4306f66ce067a49d"
server_key_version: "encryptionkey@0"
I20260927 02:42:43.213757 8125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:43.243419 8125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:43.246078 8125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:43.246243 8125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:43.246784 8125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:43.246835 8125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:43.246867 8125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:43.246884 8125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476963, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:43.253731 8125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36227
I20260927 02:42:43.253822 8255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36227 every 8 connection(s)
I20260927 02:42:43.254143 8125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:43.256508 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.247278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43353 (local address 127.0.98.2:47823)
0927 02:42:43.247426 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.247467 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.247958 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.248210 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.248214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.248543 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:43.248754 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.248759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.249689 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.249690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.250548 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.250551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.251023 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.251024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.251162 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.251646 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:43.251658 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:43.253348 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:43.255085 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.255091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.255093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.255317 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.255538 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.255539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.255539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.255653 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.255859 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:43.255870 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:43.255982 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.256109 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.256297 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260927 02:42:43.256659 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.247232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47823 (local address 127.0.98.62:43353)
0927 02:42:43.247292 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.247294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.247556 (+ 262us) server_negotiation.cc:407] Connection header received
0927 02:42:43.248047 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.248048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.248062 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.248144 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:43.248831 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.249577 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.250626 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.250908 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.250990 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.253453 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:43.253455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:43.253455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:43.253462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:43.255003 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.255387 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.255388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.255389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.255444 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.255713 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.255714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.255714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.255816 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:43.255862 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.256488 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.256567 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:42:43.256964 8256 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43353
I20260927 02:42:43.257043 8256 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:43.257248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 8125
I20260927 02:42:43.257329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:43.257513 392 external_mini_cluster.cc:1428] Setting key e43f65a2d5278a971577e4c371d5f9aa
I20260927 02:42:43.257831 8256 heartbeater.cc:511] Master 127.0.98.62:43353 requested a full tablet report, sending...
I20260927 02:42:43.258222 7913 ts_manager.cc:194] Registered new tserver with Master: f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227)
I20260927 02:42:43.258682 7913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47823
I20260927 02:42:43.263983 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.257874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43353 (local address 127.0.98.2:39539)
0927 02:42:43.258013 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.258023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.258098 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.258251 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.258252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.258438 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:43.258503 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.258504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.259344 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.259345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.260181 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.260183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.260580 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.260581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.260641 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.261027 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:43.261031 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:43.261300 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:43.262992 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.262993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.262993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.263166 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.263355 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.263356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.263356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.263409 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.263728 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:43.263729 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:43.263770 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.263835 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.263915 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260927 02:42:43.264030 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.257913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39539 (local address 127.0.98.62:43353)
0927 02:42:43.257947 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.257948 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.258074 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:42:43.258160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.258161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.258171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.258197 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:43.258576 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.259240 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.260263 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.260498 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.260563 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.261378 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:43.261379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:43.261380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:43.261386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:43.262898 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.263237 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.263238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.263238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.263297 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.263459 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.263460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.263460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.263637 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:43.263714 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.263872 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.263947 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:42:43.285213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43353
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:43.368582 8265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:43.368808 8265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:43.368839 8265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:43.370069 8265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:43.370116 8265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:43.370143 8265 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:42:43.370165 8265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:43.372032 8265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:43.372287 8265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:43.372537 8265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:43.377700 8270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:43.377636 8271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:43.377628 8273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:43.378021 8265 server_base.cc:1034] running on GCE node
I20260927 02:42:43.378250 8265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:43.378492 8265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:43.379663 8265 hybrid_clock.cc:648] HybridClock initialized: now 1790476963379649 us; error 29 us; skew 500 ppm
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476963, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:43.382287 8265 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:42:43.382694 8265 webserver.cc:460] Webserver started at http://127.0.98.3:33319/ using document root <none> and password file <none>
I20260927 02:42:43.382872 8265 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:43.382912 8265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:43.383021 8265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:43.384140 8265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "321ba40f052b42d3a1390ef59edb3bce"
format_stamp: "Formatted at 2026-09-27 02:42:43 on dist-test-slave-44kr"
server_key: "35b659d95a698dd2a6153e9d88e23343"
server_key_iv: "55c7755eb0a7c2439d3e20810a55005e"
server_key_version: "encryptionkey@0"
I20260927 02:42:43.384518 8265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "321ba40f052b42d3a1390ef59edb3bce"
format_stamp: "Formatted at 2026-09-27 02:42:43 on dist-test-slave-44kr"
server_key: "35b659d95a698dd2a6153e9d88e23343"
server_key_iv: "55c7755eb0a7c2439d3e20810a55005e"
server_key_version: "encryptionkey@0"
I20260927 02:42:43.385866 8265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:43.386595 8280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:43.386776 8265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:42:43.386849 8265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "321ba40f052b42d3a1390ef59edb3bce"
format_stamp: "Formatted at 2026-09-27 02:42:43 on dist-test-slave-44kr"
server_key: "35b659d95a698dd2a6153e9d88e23343"
server_key_iv: "55c7755eb0a7c2439d3e20810a55005e"
server_key_version: "encryptionkey@0"
I20260927 02:42:43.386904 8265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:43.407150 8265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:43.409294 8265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:43.409407 8265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:43.409873 8265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:43.409914 8265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:43.409943 8265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:43.409965 8265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:43.416977 8265 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43057
I20260927 02:42:43.417053 8395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43057 every 8 connection(s)
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476963, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:43.417346 8265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:43.420189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 8265
I20260927 02:42:43.420284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:43.420476 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.410442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39051 (local address 127.0.98.62:43353)
0927 02:42:43.410488 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.410490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.410754 (+ 264us) server_negotiation.cc:407] Connection header received
0927 02:42:43.411312 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.411314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.411329 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.411399 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:43.412305 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.413110 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.414678 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.415033 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.415091 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.417847 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:43.417849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:43.417849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:43.417856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:43.419080 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.419491 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.419491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.419492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.419537 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.419753 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.419753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.419754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.419849 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:43.419893 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.420200 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.420348 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:43.420500 392 external_mini_cluster.cc:1428] Setting key 1f9c73f37043a7f88c3f14b7a2c81969
I20260927 02:42:43.420584 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.410449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43353 (local address 127.0.98.3:39051)
0927 02:42:43.410605 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.410643 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.411151 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.411520 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.411525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.411910 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:43.412211 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.412218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.413208 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.413210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.414578 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.414581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.415120 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.415121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.415319 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.415939 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:43.415950 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:43.417770 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:43.419185 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.419191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.419200 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.419422 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.419607 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.419608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.419608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.419707 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.419899 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:43.419903 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:43.420043 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.420161 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.420318 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260927 02:42:43.421051 8396 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43353
I20260927 02:42:43.421134 8396 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:43.421315 8396 heartbeater.cc:511] Master 127.0.98.62:43353 requested a full tablet report, sending...
I20260927 02:42:43.421736 7912 ts_manager.cc:194] Registered new tserver with Master: 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057)
I20260927 02:42:43.422205 7912 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39051
I20260927 02:42:43.427197 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.422087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33361 (local address 127.0.98.62:43353)
0927 02:42:43.422339 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.422341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.422349 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:43.422367 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.422367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.422377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.422433 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:43.422793 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.423257 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.423964 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.424138 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.424189 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.425136 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:43.425137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:43.425138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:43.425143 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:43.426367 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.426672 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.426672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.426673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.426730 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.426863 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.426863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.426864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.426949 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:43.426990 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.427085 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.427138 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260927 02:42:43.427215 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.422029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43353 (local address 127.0.98.3:33361)
0927 02:42:43.422144 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.422154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.422199 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.422444 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.422445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.422625 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:43.422722 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.422724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.423358 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.423359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.423895 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.423897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.424256 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.424257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.424312 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.424714 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:43.424718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:43.424997 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:43.426442 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.426443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.426444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.426614 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.426776 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:43.426777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:43.426777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:43.426821 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:43.426988 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:43.426988 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:43.427020 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.427069 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.427140 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":5,"thread_start_us":65,"threads_started":1}
I20260927 02:42:43.432222 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476963, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:42:43.452881 7980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:43.447546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.0.98.62:43353)
0927 02:42:43.447612 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.447614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.447622 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:43.447647 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.447648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.447656 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.447702 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:43.448244 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.448783 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.449501 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.449682 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.449735 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.452809 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54136: BlockingRecv error: recv got EOF from 127.0.0.1:54136 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 02:42:43 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476963, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:43.460630 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.453584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.0.98.62:43353)
0927 02:42:43.453644 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.453645 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.453654 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:43.453710 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.453710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.453720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.453797 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:43.454214 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.454728 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.455559 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.455818 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.455875 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.457991 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:43.457992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:43.457992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:43.457999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:43.459300 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.459737 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.459739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.459739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.459797 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:43.460045 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:43.460046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:43.460046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:43.460168 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:43.460255 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.460457 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.460518 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:42:43.462333 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:43.463825 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:43.470108 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.466072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36227 (local address 127.0.0.1:54996)
0927 02:42:43.466253 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.466296 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.466361 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.466867 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.466868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.466870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:43.466923 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.466924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.467850 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.467851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.469019 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.469022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.469808 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.469809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.469894 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.469908 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.469971 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.470021 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":57,"threads_started":1}
I20260927 02:42:43.470124 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.466251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.0.98.2:36227)
0927 02:42:43.466717 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.466720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.466728 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:43.466765 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.466767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.466783 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.466858 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:43.467095 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.467774 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.469100 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.469734 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.469796 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.469876 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.470001 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.470036 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":89,"threads_started":1}
I20260927 02:42:43.470481 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.466105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43057 (local address 127.0.0.1:55920)
0927 02:42:43.466479 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.466489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.466553 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.466781 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.466785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.466787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:43.466873 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.466875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.468244 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.468245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.469317 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.469319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.470261 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.470262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.470325 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.470328 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.470376 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.470414 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":92,"thread_start_us":84,"threads_started":1}
I20260927 02:42:43.470531 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.466322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.0.98.3:43057)
0927 02:42:43.466490 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.466495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.466539 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:42:43.466664 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.466667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.466684 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.466976 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:43.467039 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.468166 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.469395 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.470082 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.470149 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.470223 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.470394 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.470438 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260927 02:42:43.470836 8190 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6f13c937df4f4c87d91499ed6eedeb (DEFAULT_TABLE table=test-table [id=2667c16582204b1c8b1103340239193e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:43.471035 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.466343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.0.98.1:46107)
0927 02:42:43.466507 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.466511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.466624 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:42:43.466671 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.466673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.466687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.466766 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:43.467024 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.468147 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.469370 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.470078 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.470149 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.470819 (+ 670us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.470905 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.470942 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260927 02:42:43.470983 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.466136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.0.1:58284)
0927 02:42:43.466546 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.466555 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.466605 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.466935 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.466936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.466937 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:43.466967 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.466968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.468250 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.468251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.469290 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.469292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.470464 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.470465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.470516 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.470519 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.470882 (+ 363us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.470921 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":29,"thread_start_us":78,"threads_started":1}
I20260927 02:42:43.471232 8190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6f13c937df4f4c87d91499ed6eedeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:43.471197 8330 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6f13c937df4f4c87d91499ed6eedeb (DEFAULT_TABLE table=test-table [id=2667c16582204b1c8b1103340239193e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:43.471516 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6f13c937df4f4c87d91499ed6eedeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:43.471805 8050 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6f13c937df4f4c87d91499ed6eedeb (DEFAULT_TABLE table=test-table [id=2667c16582204b1c8b1103340239193e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:43.472077 8050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6f13c937df4f4c87d91499ed6eedeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:43.474997 8424 tablet_bootstrap.cc:492] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: Bootstrap starting.
I20260927 02:42:43.475381 8426 tablet_bootstrap.cc:492] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: Bootstrap starting.
I20260927 02:42:43.476070 8426 tablet_bootstrap.cc:654] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:43.476130 8424 tablet_bootstrap.cc:654] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:43.476493 8424 log.cc:824] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:43.476508 8426 log.cc:824] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:43.476580 8425 tablet_bootstrap.cc:492] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: Bootstrap starting.
I20260927 02:42:43.477121 8425 tablet_bootstrap.cc:654] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:43.477216 8426 tablet_bootstrap.cc:492] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: No bootstrap required, opened a new log
I20260927 02:42:43.477284 8424 tablet_bootstrap.cc:492] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: No bootstrap required, opened a new log
I20260927 02:42:43.477293 8426 ts_tablet_manager.cc:1397] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:43.477352 8424 ts_tablet_manager.cc:1397] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:42:43.477437 8425 log.cc:824] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:43.478533 8425 tablet_bootstrap.cc:492] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: No bootstrap required, opened a new log
I20260927 02:42:43.478607 8425 ts_tablet_manager.cc:1397] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:43.479099 8426 raft_consensus.cc:348] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.479245 8426 raft_consensus.cc:374] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:43.479290 8426 raft_consensus.cc:729] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63b020da5194b2e95d5d8403bba5b69, State: Initialized, Role: FOLLOWER
I20260927 02:42:43.479283 8424 raft_consensus.cc:348] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.479458 8424 raft_consensus.cc:374] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:43.479420 8426 consensus_queue.cc:260] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.479502 8424 raft_consensus.cc:729] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d50c57e6ed64cd9a8c159569e28e610, State: Initialized, Role: FOLLOWER
I20260927 02:42:43.479651 8424 consensus_queue.cc:260] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.479712 8426 ts_tablet_manager.cc:1428] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:43.479825 8256 heartbeater.cc:503] Master 127.0.98.62:43353 was elected leader, sending a full tablet report...
I20260927 02:42:43.479971 8116 heartbeater.cc:503] Master 127.0.98.62:43353 was elected leader, sending a full tablet report...
I20260927 02:42:43.479918 8424 ts_tablet_manager.cc:1428] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 02:42:43.480682 8425 raft_consensus.cc:348] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.480835 8425 raft_consensus.cc:374] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:43.480871 8425 raft_consensus.cc:729] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321ba40f052b42d3a1390ef59edb3bce, State: Initialized, Role: FOLLOWER
I20260927 02:42:43.481004 8425 consensus_queue.cc:260] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.481292 8396 heartbeater.cc:503] Master 127.0.98.62:43353 was elected leader, sending a full tablet report...
I20260927 02:42:43.481276 8425 ts_tablet_manager.cc:1428] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260927 02:42:43.504777 8257 tablet.cc:2369] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:42:43.579388 8117 tablet.cc:2369] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:42:43.667917 8397 tablet.cc:2369] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:43.699033 8431 raft_consensus.cc:482] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:43.699157 8431 raft_consensus.cc:504] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.699535 8431 leader_election.cc:288] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227), 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057)
I20260927 02:42:43.702405 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.699807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35471 (local address 127.0.98.2:36227)
0927 02:42:43.699858 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.699860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.699909 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:42:43.699996 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.699997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.700008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.700057 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:43.700198 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.700709 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.701748 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.702095 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.702157 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.702168 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.702305 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.702339 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:42:43.702409 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.699704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36227 (local address 127.0.98.1:35471)
0927 02:42:43.699853 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.699863 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.699925 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.700071 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.700071 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.700073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:43.700138 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.700140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.700778 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.700779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.701663 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.701666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.702170 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.702171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.702243 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.702246 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.702290 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.702331 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260927 02:42:43.702540 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.699894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44023 (local address 127.0.98.3:43057)
0927 02:42:43.699935 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.699937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.699946 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:43.699996 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.699997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.700007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.700052 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:43.700236 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.700750 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.701731 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.702074 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.702207 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.702221 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.702395 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.702443 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:43.702772 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.699738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43057 (local address 127.0.98.1:44023)
0927 02:42:43.699867 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.699879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.699933 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.700070 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.700070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.700072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:43.700165 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.700167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.700834 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.700834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.701652 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.701655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.702207 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.702208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.702299 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.702304 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.702629 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.702678 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260927 02:42:43.702983 8210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f13c937df4f4c87d91499ed6eedeb" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69" is_pre_election: true
I20260927 02:42:43.703178 8210 raft_consensus.cc:2454] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 0.
I20260927 02:42:43.703428 8002 leader_election.cc:302] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d50c57e6ed64cd9a8c159569e28e610, f63b020da5194b2e95d5d8403bba5b69; no voters:
I20260927 02:42:43.703575 8431 raft_consensus.cc:2781] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:43.703634 8431 raft_consensus.cc:482] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:43.703655 8431 raft_consensus.cc:3037] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:43.703765 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f13c937df4f4c87d91499ed6eedeb" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321ba40f052b42d3a1390ef59edb3bce" is_pre_election: true
I20260927 02:42:43.703910 8350 raft_consensus.cc:2454] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 0.
I20260927 02:42:43.704506 8431 raft_consensus.cc:504] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.704630 8431 leader_election.cc:288] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 election: Requested vote from peers f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227), 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057)
I20260927 02:42:43.704794 8210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f13c937df4f4c87d91499ed6eedeb" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69"
I20260927 02:42:43.704841 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f13c937df4f4c87d91499ed6eedeb" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321ba40f052b42d3a1390ef59edb3bce"
I20260927 02:42:43.704878 8210 raft_consensus.cc:3037] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:43.704917 8350 raft_consensus.cc:3037] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:43.705561 8350 raft_consensus.cc:2454] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 1.
I20260927 02:42:43.705646 8210 raft_consensus.cc:2454] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 1.
I20260927 02:42:43.705762 8003 leader_election.cc:302] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321ba40f052b42d3a1390ef59edb3bce, 8d50c57e6ed64cd9a8c159569e28e610; no voters:
I20260927 02:42:43.705865 8431 raft_consensus.cc:2781] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:43.705957 8431 raft_consensus.cc:686] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 LEADER]: Becoming Leader. State: Replica: 8d50c57e6ed64cd9a8c159569e28e610, State: Running, Role: LEADER
I20260927 02:42:43.706039 8431 consensus_queue.cc:237] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.706802 7913 catalog_manager.cc:5495] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8d50c57e6ed64cd9a8c159569e28e610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:43.758119 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.755508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.0.98.1:46107)
0927 02:42:43.755563 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.755564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.755573 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:43.755638 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.755639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.755658 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.755726 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:43.755847 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.756459 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.757341 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.757556 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.757608 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.757810 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:43.757920 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:43.757978 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.758020 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.758038 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:42:43.760100 8210 raft_consensus.cc:1261] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Refusing update from remote peer 8d50c57e6ed64cd9a8c159569e28e610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:43.760162 8350 raft_consensus.cc:1261] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Refusing update from remote peer 8d50c57e6ed64cd9a8c159569e28e610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:43.760458 8431 consensus_queue.cc:1035] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:43.760620 8435 consensus_queue.cc:1035] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:43.762930 8438 mvcc.cc:205] Tried to move back new op lower bound from 7333793643558600704 to 7333793643340550144. Current Snapshot: MvccSnapshot[applied={T|T < 7333793643558600704}]
I20260927 02:42:43.763779 8440 mvcc.cc:205] Tried to move back new op lower bound from 7333793643558600704 to 7333793643340550144. Current Snapshot: MvccSnapshot[applied={T|T < 7333793643558600704 or (T in {7333793643558600704})}]
I20260927 02:42:43.765380 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:43.765481 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:43.766197 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: Sending DeleteTablet for 3 replicas of tablet 2f6f13c937df4f4c87d91499ed6eedeb
I20260927 02:42:43.766549 8190 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6f13c937df4f4c87d91499ed6eedeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54996
I20260927 02:42:43.766556 8050 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6f13c937df4f4c87d91499ed6eedeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260927 02:42:43.766554 8330 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6f13c937df4f4c87d91499ed6eedeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55920
I20260927 02:42:43.766855 8448 tablet_replica.cc:330] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: stopping tablet replica
I20260927 02:42:43.766990 8448 raft_consensus.cc:2229] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:43.767041 8449 tablet_replica.cc:330] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: stopping tablet replica
I20260927 02:42:43.767006 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:43.767130 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:43.767187 8448 raft_consensus.cc:2258] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:43.767369 8449 raft_consensus.cc:2229] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:43.767544 8449 raft_consensus.cc:2258] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:43.767661 8448 ts_tablet_manager.cc:1905] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:43.768052 8449 ts_tablet_manager.cc:1905] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:43.768355 8330 tablet_service.cc:1467] Processing CreateTablet for tablet 739650a58af4408487065aed961fdd36 (DEFAULT_TABLE table=test-table [id=1e525bc239ef4cf0809dab5ecc135835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:43.768446 8190 tablet_service.cc:1467] Processing CreateTablet for tablet 739650a58af4408487065aed961fdd36 (DEFAULT_TABLE table=test-table [id=1e525bc239ef4cf0809dab5ecc135835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:43.768647 8190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739650a58af4408487065aed961fdd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:43.768672 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739650a58af4408487065aed961fdd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:43.768764 8450 tablet_replica.cc:330] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: stopping tablet replica
I20260927 02:42:43.768847 8450 raft_consensus.cc:2229] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:43.768929 8450 raft_consensus.cc:2258] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:43.769559 8448 ts_tablet_manager.cc:1918] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:43.769618 8448 log.cc:1196] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/2f6f13c937df4f4c87d91499ed6eedeb
I20260927 02:42:43.769845 8448 ts_tablet_manager.cc:1939] T 2f6f13c937df4f4c87d91499ed6eedeb P f63b020da5194b2e95d5d8403bba5b69: Deleting consensus metadata
I20260927 02:42:43.769850 8426 tablet_bootstrap.cc:492] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: Bootstrap starting.
I20260927 02:42:43.769948 8425 tablet_bootstrap.cc:492] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: Bootstrap starting.
I20260927 02:42:43.770242 8426 tablet_bootstrap.cc:654] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:43.770308 8450 ts_tablet_manager.cc:1905] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:43.770357 8425 tablet_bootstrap.cc:654] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:43.770787 8050 tablet_service.cc:1467] Processing CreateTablet for tablet 739650a58af4408487065aed961fdd36 (DEFAULT_TABLE table=test-table [id=1e525bc239ef4cf0809dab5ecc135835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:43.770887 8050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739650a58af4408487065aed961fdd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:43.770934 7898 catalog_manager.cc:4841] TS f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227): tablet 2f6f13c937df4f4c87d91499ed6eedeb (table test-table [id=2667c16582204b1c8b1103340239193e]) successfully deleted
I20260927 02:42:43.771634 8449 ts_tablet_manager.cc:1918] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:43.771700 8449 log.cc:1196] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/2f6f13c937df4f4c87d91499ed6eedeb
I20260927 02:42:43.771734 8450 ts_tablet_manager.cc:1918] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:43.771792 8450 log.cc:1196] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/2f6f13c937df4f4c87d91499ed6eedeb
I20260927 02:42:43.771943 8449 ts_tablet_manager.cc:1939] T 2f6f13c937df4f4c87d91499ed6eedeb P 321ba40f052b42d3a1390ef59edb3bce: Deleting consensus metadata
I20260927 02:42:43.772032 8424 tablet_bootstrap.cc:492] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: Bootstrap starting.
I20260927 02:42:43.772475 8424 tablet_bootstrap.cc:654] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:43.772616 7899 catalog_manager.cc:4841] TS 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057): tablet 2f6f13c937df4f4c87d91499ed6eedeb (table test-table [id=2667c16582204b1c8b1103340239193e]) successfully deleted
I20260927 02:42:43.772022 8450 ts_tablet_manager.cc:1939] T 2f6f13c937df4f4c87d91499ed6eedeb P 8d50c57e6ed64cd9a8c159569e28e610: Deleting consensus metadata
I20260927 02:42:43.772936 8425 tablet_bootstrap.cc:492] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: No bootstrap required, opened a new log
I20260927 02:42:43.772951 8426 tablet_bootstrap.cc:492] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: No bootstrap required, opened a new log
I20260927 02:42:43.772984 8425 ts_tablet_manager.cc:1397] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:42:43.772992 8426 ts_tablet_manager.cc:1397] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:42:43.773129 8425 raft_consensus.cc:348] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.773195 8425 raft_consensus.cc:374] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:43.773214 8425 raft_consensus.cc:729] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321ba40f052b42d3a1390ef59edb3bce, State: Initialized, Role: FOLLOWER
I20260927 02:42:43.773260 8425 consensus_queue.cc:260] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.773334 8425 ts_tablet_manager.cc:1428] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:43.773346 7897 catalog_manager.cc:4841] TS 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1:46107): tablet 2f6f13c937df4f4c87d91499ed6eedeb (table test-table [id=2667c16582204b1c8b1103340239193e]) successfully deleted
I20260927 02:42:43.773412 8426 raft_consensus.cc:348] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.773481 8426 raft_consensus.cc:374] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:43.773499 8426 raft_consensus.cc:729] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63b020da5194b2e95d5d8403bba5b69, State: Initialized, Role: FOLLOWER
I20260927 02:42:43.773541 8426 consensus_queue.cc:260] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.773629 8426 ts_tablet_manager.cc:1428] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:42:43.774015 8424 tablet_bootstrap.cc:492] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: No bootstrap required, opened a new log
I20260927 02:42:43.774065 8424 ts_tablet_manager.cc:1397] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:42:43.774204 8424 raft_consensus.cc:348] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.774294 8424 raft_consensus.cc:374] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:43.774317 8424 raft_consensus.cc:729] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d50c57e6ed64cd9a8c159569e28e610, State: Initialized, Role: FOLLOWER
I20260927 02:42:43.774353 8424 consensus_queue.cc:260] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.774433 8424 ts_tablet_manager.cc:1428] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:43.986251 8432 raft_consensus.cc:482] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:43.986356 8432 raft_consensus.cc:504] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.986688 8432 leader_election.cc:288] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1:46107), f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227)
I20260927 02:42:43.989666 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.986931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34653 (local address 127.0.98.1:46107)
0927 02:42:43.986989 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.986991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.987083 (+ 92us) server_negotiation.cc:407] Connection header received
0927 02:42:43.987153 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.987154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.987162 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.987263 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:43.987413 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.987884 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.988971 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.989340 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.989400 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.989411 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.989551 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.989587 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:42:43.989681 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.986877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.98.3:34653)
0927 02:42:43.987007 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.987018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.987096 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.987278 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.987279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.987281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:43.987354 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.987357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.987956 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.987957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.988886 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.988889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.989412 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.989413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.989494 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.989497 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.989541 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.989588 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260927 02:42:43.989804 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.986940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56905 (local address 127.0.98.2:36227)
0927 02:42:43.986989 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.986991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.987205 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:42:43.987235 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.987236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.987243 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.987288 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:43.987424 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.987912 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.989024 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.989488 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.989549 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.989558 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.989699 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.989735 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:42:43.989817 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.986877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36227 (local address 127.0.98.3:56905)
0927 02:42:43.987090 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.987100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:43.987165 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:43.987296 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:43.987296 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:43.987298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:43.987367 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.987369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.987987 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.987988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.988936 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:43.988939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.989556 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.989557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:43.989641 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.989644 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:43.989688 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:42:43.989731 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":148,"thread_start_us":39,"threads_started":1}
I20260927 02:42:43.990109 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739650a58af4408487065aed961fdd36" candidate_uuid: "321ba40f052b42d3a1390ef59edb3bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d50c57e6ed64cd9a8c159569e28e610" is_pre_election: true
I20260927 02:42:43.990269 8070 raft_consensus.cc:2454] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 321ba40f052b42d3a1390ef59edb3bce in term 0.
I20260927 02:42:43.990293 8210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739650a58af4408487065aed961fdd36" candidate_uuid: "321ba40f052b42d3a1390ef59edb3bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69" is_pre_election: true
I20260927 02:42:43.990366 8210 raft_consensus.cc:2454] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 321ba40f052b42d3a1390ef59edb3bce in term 0.
I20260927 02:42:43.990459 8281 leader_election.cc:302] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321ba40f052b42d3a1390ef59edb3bce, 8d50c57e6ed64cd9a8c159569e28e610; no voters:
I20260927 02:42:43.990592 8432 raft_consensus.cc:2781] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:43.990628 8432 raft_consensus.cc:482] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:43.990648 8432 raft_consensus.cc:3037] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:43.991271 8432 raft_consensus.cc:504] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.991403 8432 leader_election.cc:288] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [CANDIDATE]: Term 1 election: Requested vote from peers 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1:46107), f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227)
I20260927 02:42:43.991554 8210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739650a58af4408487065aed961fdd36" candidate_uuid: "321ba40f052b42d3a1390ef59edb3bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69"
I20260927 02:42:43.991593 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739650a58af4408487065aed961fdd36" candidate_uuid: "321ba40f052b42d3a1390ef59edb3bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d50c57e6ed64cd9a8c159569e28e610"
I20260927 02:42:43.991652 8210 raft_consensus.cc:3037] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:43.991657 8070 raft_consensus.cc:3037] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:43.992482 8210 raft_consensus.cc:2454] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 321ba40f052b42d3a1390ef59edb3bce in term 1.
I20260927 02:42:43.992482 8070 raft_consensus.cc:2454] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 321ba40f052b42d3a1390ef59edb3bce in term 1.
I20260927 02:42:43.992646 8281 leader_election.cc:302] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 321ba40f052b42d3a1390ef59edb3bce, 8d50c57e6ed64cd9a8c159569e28e610, f63b020da5194b2e95d5d8403bba5b69; no voters:
I20260927 02:42:43.992751 8432 raft_consensus.cc:2781] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:43.992868 8432 raft_consensus.cc:686] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 1 LEADER]: Becoming Leader. State: Replica: 321ba40f052b42d3a1390ef59edb3bce, State: Running, Role: LEADER
I20260927 02:42:43.992934 8432 consensus_queue.cc:237] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:43.993556 7912 catalog_manager.cc:5495] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce reported cstate change: term changed from 0 to 1, leader changed from <none> to 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3). New cstate: current_term: 1 leader_uuid: "321ba40f052b42d3a1390ef59edb3bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } health_report { overall_health: HEALTHY } } }
Sep 27 02:42:44 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:44.044574 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.037214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.0.98.62:43353)
0927 02:42:44.037352 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.037354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.037401 (+ 47us) server_negotiation.cc:407] Connection header received
0927 02:42:44.037455 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.037456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.037465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.037525 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:44.038099 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.038656 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.039408 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.039613 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.039679 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.041891 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:44.041892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:44.041892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:44.041899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:44.043369 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.043802 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.043803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.043804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.043866 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.044025 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.044026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.044026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.044122 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:44.044170 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.044365 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.044491 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260927 02:42:44.044697 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.037140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43353 (local address 127.0.0.1:54148)
0927 02:42:44.037324 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:44.037335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:44.037412 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:44.037596 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:44.037597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:44.037925 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:44.037992 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.037994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.038750 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.038751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.039315 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.039316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.039712 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.039713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.039782 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.040319 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:44.040323 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:44.041805 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:44.043438 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:44.043444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:44.043447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:44.043726 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:44.043920 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:44.043921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:44.043921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:44.043986 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:44.044219 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:44.044225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:44.044285 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 02:42:44.044349 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:42:44.044597 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":86,"thread_start_us":82,"threads_started":1}
I20260927 02:42:44.046535 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54148:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:42:44.046725 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:44.048045 8050 tablet_service.cc:1467] Processing CreateTablet for tablet ca904c111c2d4d2fa37edca739d97f5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=525e0db6e0d44ce28710bdf13c59f655]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:42:44.048060 8190 tablet_service.cc:1467] Processing CreateTablet for tablet ca904c111c2d4d2fa37edca739d97f5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=525e0db6e0d44ce28710bdf13c59f655]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:42:44.048187 8050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca904c111c2d4d2fa37edca739d97f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.048238 8190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca904c111c2d4d2fa37edca739d97f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.049373 8426 tablet_bootstrap.cc:492] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: Bootstrap starting.
I20260927 02:42:44.049451 8424 tablet_bootstrap.cc:492] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: Bootstrap starting.
I20260927 02:42:44.049543 8330 tablet_service.cc:1467] Processing CreateTablet for tablet ca904c111c2d4d2fa37edca739d97f5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=525e0db6e0d44ce28710bdf13c59f655]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:42:44.049710 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca904c111c2d4d2fa37edca739d97f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.049723 8424 tablet_bootstrap.cc:654] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:44.049785 8426 tablet_bootstrap.cc:654] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:44.050202 8424 tablet_bootstrap.cc:492] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: No bootstrap required, opened a new log
I20260927 02:42:44.050246 8424 ts_tablet_manager.cc:1397] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:42:44.050367 8424 raft_consensus.cc:348] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.050423 8424 raft_consensus.cc:374] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.050441 8424 raft_consensus.cc:729] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d50c57e6ed64cd9a8c159569e28e610, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.050482 8424 consensus_queue.cc:260] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.050572 8424 ts_tablet_manager.cc:1428] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.050608 8435 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.050700 8435 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:44.050971 8426 tablet_bootstrap.cc:492] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: No bootstrap required, opened a new log
I20260927 02:42:44.051023 8426 ts_tablet_manager.cc:1397] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:44.051108 8425 tablet_bootstrap.cc:492] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: Bootstrap starting.
I20260927 02:42:44.051159 8426 raft_consensus.cc:348] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.051221 8426 raft_consensus.cc:374] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.051240 8426 raft_consensus.cc:729] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63b020da5194b2e95d5d8403bba5b69, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.051281 8426 consensus_queue.cc:260] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.051369 8426 ts_tablet_manager.cc:1428] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.051414 8430 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.051487 8430 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:44.051520 8425 tablet_bootstrap.cc:654] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:44.052168 8425 tablet_bootstrap.cc:492] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: No bootstrap required, opened a new log
I20260927 02:42:44.052213 8425 ts_tablet_manager.cc:1397] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:44.052325 8425 raft_consensus.cc:348] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.052429 8425 raft_consensus.cc:374] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.052456 8425 raft_consensus.cc:729] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321ba40f052b42d3a1390ef59edb3bce, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.052505 8425 consensus_queue.cc:260] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.052591 8425 ts_tablet_manager.cc:1428] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.052639 8432 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.052731 8432 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:44.215206 8432 raft_consensus.cc:482] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:44.215313 8432 raft_consensus.cc:504] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.215477 8432 leader_election.cc:288] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1:46107), f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227)
I20260927 02:42:44.215735 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca904c111c2d4d2fa37edca739d97f5c" candidate_uuid: "321ba40f052b42d3a1390ef59edb3bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d50c57e6ed64cd9a8c159569e28e610" is_pre_election: true
I20260927 02:42:44.215832 8070 raft_consensus.cc:2454] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 321ba40f052b42d3a1390ef59edb3bce in term 0.
I20260927 02:42:44.215807 8210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca904c111c2d4d2fa37edca739d97f5c" candidate_uuid: "321ba40f052b42d3a1390ef59edb3bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69" is_pre_election: true
I20260927 02:42:44.215912 8210 raft_consensus.cc:2454] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 321ba40f052b42d3a1390ef59edb3bce in term 0.
I20260927 02:42:44.215978 8281 leader_election.cc:302] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321ba40f052b42d3a1390ef59edb3bce, 8d50c57e6ed64cd9a8c159569e28e610; no voters:
I20260927 02:42:44.216187 8432 raft_consensus.cc:2781] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:44.216255 8432 raft_consensus.cc:482] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:44.216269 8432 raft_consensus.cc:3037] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.216995 8432 raft_consensus.cc:504] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.217127 8432 leader_election.cc:288] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [CANDIDATE]: Term 1 election: Requested vote from peers 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1:46107), f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227)
I20260927 02:42:44.217309 8210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca904c111c2d4d2fa37edca739d97f5c" candidate_uuid: "321ba40f052b42d3a1390ef59edb3bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69"
I20260927 02:42:44.217361 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca904c111c2d4d2fa37edca739d97f5c" candidate_uuid: "321ba40f052b42d3a1390ef59edb3bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d50c57e6ed64cd9a8c159569e28e610"
I20260927 02:42:44.217401 8210 raft_consensus.cc:3037] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.217433 8070 raft_consensus.cc:3037] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.218189 8070 raft_consensus.cc:2454] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 321ba40f052b42d3a1390ef59edb3bce in term 1.
I20260927 02:42:44.218189 8210 raft_consensus.cc:2454] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 321ba40f052b42d3a1390ef59edb3bce in term 1.
I20260927 02:42:44.218377 8282 leader_election.cc:302] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321ba40f052b42d3a1390ef59edb3bce, f63b020da5194b2e95d5d8403bba5b69; no voters:
I20260927 02:42:44.218482 8432 raft_consensus.cc:2781] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:44.218529 8432 raft_consensus.cc:686] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [term 1 LEADER]: Becoming Leader. State: Replica: 321ba40f052b42d3a1390ef59edb3bce, State: Running, Role: LEADER
I20260927 02:42:44.218583 8432 consensus_queue.cc:237] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.218855 8456 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: TxnStatusTablet state changed. Reason: New leader 321ba40f052b42d3a1390ef59edb3bce. Latest consensus state: current_term: 1 leader_uuid: "321ba40f052b42d3a1390ef59edb3bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.218942 8456 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:42:44.219138 8469 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:42:44.219170 7913 catalog_manager.cc:5495] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce reported cstate change: term changed from 0 to 1, leader changed from <none> to 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3). New cstate: current_term: 1 leader_uuid: "321ba40f052b42d3a1390ef59edb3bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } health_report { overall_health: HEALTHY } } }
I20260927 02:42:44.219267 8469 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:42:44.236940 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.233786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.0.98.3:43057)
0927 02:42:44.233990 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.233992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.234001 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:44.234040 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.234041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.234071 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.234148 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:44.234310 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.234829 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.235967 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.236234 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.236306 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.236527 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.236658 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.236736 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.236789 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.236816 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260927 02:42:44.236934 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.233435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43057 (local address 127.0.0.1:55924)
0927 02:42:44.233518 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:44.233533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:44.233604 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:44.234176 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:44.234178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:44.234180 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:44.234237 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.234240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.234894 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.234895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.235866 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.235870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.236311 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.236312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.236375 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.236422 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.236729 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.236738 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 02:42:44.236784 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:42:44.236844 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:42:44.238168 8210 raft_consensus.cc:1261] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Refusing update from remote peer 321ba40f052b42d3a1390ef59edb3bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:44.238193 8070 raft_consensus.cc:1261] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Refusing update from remote peer 321ba40f052b42d3a1390ef59edb3bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:44.238463 8432 consensus_queue.cc:1035] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:44.238595 8456 consensus_queue.cc:1035] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:44.239239 8430 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: TxnStatusTablet state changed. Reason: New leader 321ba40f052b42d3a1390ef59edb3bce. Latest consensus state: current_term: 1 leader_uuid: "321ba40f052b42d3a1390ef59edb3bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.239333 8430 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:44.239409 8435 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: TxnStatusTablet state changed. Reason: New leader 321ba40f052b42d3a1390ef59edb3bce. Latest consensus state: current_term: 1 leader_uuid: "321ba40f052b42d3a1390ef59edb3bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.239494 8435 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:44.240219 8456 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "321ba40f052b42d3a1390ef59edb3bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.240305 8456 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:42:44.240418 8439 mvcc.cc:205] Tried to move back new op lower bound from 7333793645517160448 to 7333793645439672320. Current Snapshot: MvccSnapshot[applied={T|T < 7333793645517160448}]
I20260927 02:42:44.240522 8430 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "321ba40f052b42d3a1390ef59edb3bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.240602 8430 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P f63b020da5194b2e95d5d8403bba5b69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:44.241114 8432 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "321ba40f052b42d3a1390ef59edb3bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.241201 8432 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P 321ba40f052b42d3a1390ef59edb3bce: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:42:44.241931 8435 tablet_replica.cc:439] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "321ba40f052b42d3a1390ef59edb3bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } } }
I20260927 02:42:44.242025 8435 tablet_replica.cc:442] T ca904c111c2d4d2fa37edca739d97f5c P 8d50c57e6ed64cd9a8c159569e28e610: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:42:44.244733 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.242537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.0.98.3:43057)
0927 02:42:44.242594 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.242596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.242602 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:42:44.242623 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.242624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.242633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.242685 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:44.242784 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.243333 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.244067 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.244273 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.244327 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.244489 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.244535 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.244577 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.244636 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.244662 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:42:44.250362 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.247611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48381 (local address 127.0.98.3:43057)
0927 02:42:44.247653 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.247655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.247781 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:42:44.247841 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.247842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.247849 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.247889 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:44.248025 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.248566 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.249682 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.249874 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.249933 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.250104 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.250129 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.250161 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.250260 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.250277 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:42:44.250430 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.247535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43057 (local address 127.0.98.3:48381)
0927 02:42:44.247709 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:44.247721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:44.247785 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:44.247902 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:44.247902 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:44.247904 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:44.247975 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.247976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.248637 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.248637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.249601 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.249604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.249946 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.249947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.250005 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.250024 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.250175 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.250176 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:42:44.250217 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 02:42:44.250260 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260927 02:42:44.252856 8210 raft_consensus.cc:1261] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Refusing update from remote peer 321ba40f052b42d3a1390ef59edb3bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:44.252893 8070 raft_consensus.cc:1261] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Refusing update from remote peer 321ba40f052b42d3a1390ef59edb3bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:44.253036 8432 consensus_queue.cc:1035] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:44.253149 8432 consensus_queue.cc:1035] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:44.265334 7913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:44.265453 7913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:44.266047 7913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: Sending DeleteTablet for 3 replicas of tablet 739650a58af4408487065aed961fdd36
I20260927 02:42:44.266284 8050 tablet_service.cc:1514] Processing DeleteTablet for tablet 739650a58af4408487065aed961fdd36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260927 02:42:44.266362 8190 tablet_service.cc:1514] Processing DeleteTablet for tablet 739650a58af4408487065aed961fdd36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54996
I20260927 02:42:44.266393 8450 tablet_replica.cc:330] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: stopping tablet replica
I20260927 02:42:44.266389 8330 tablet_service.cc:1514] Processing DeleteTablet for tablet 739650a58af4408487065aed961fdd36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55920
I20260927 02:42:44.266448 8450 raft_consensus.cc:2229] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:44.266472 8449 tablet_replica.cc:330] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: stopping tablet replica
I20260927 02:42:44.266484 8450 raft_consensus.cc:2258] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:44.266481 8448 tablet_replica.cc:330] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: stopping tablet replica
I20260927 02:42:44.266526 8449 raft_consensus.cc:2229] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:44.266526 8448 raft_consensus.cc:2229] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:44.266559 8448 raft_consensus.cc:2258] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:44.266607 8449 raft_consensus.cc:2258] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:44.266916 8449 ts_tablet_manager.cc:1905] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:44.266917 8448 ts_tablet_manager.cc:1905] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:44.266930 8450 ts_tablet_manager.cc:1905] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:44.268379 8450 ts_tablet_manager.cc:1918] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:44.268450 8450 log.cc:1196] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/739650a58af4408487065aed961fdd36
I20260927 02:42:44.268468 8449 ts_tablet_manager.cc:1918] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:44.268471 8448 ts_tablet_manager.cc:1918] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:44.268517 8448 log.cc:1196] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/739650a58af4408487065aed961fdd36
I20260927 02:42:44.268517 8449 log.cc:1196] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/739650a58af4408487065aed961fdd36
I20260927 02:42:44.268646 8450 ts_tablet_manager.cc:1939] T 739650a58af4408487065aed961fdd36 P 8d50c57e6ed64cd9a8c159569e28e610: Deleting consensus metadata
I20260927 02:42:44.268724 8448 ts_tablet_manager.cc:1939] T 739650a58af4408487065aed961fdd36 P f63b020da5194b2e95d5d8403bba5b69: Deleting consensus metadata
I20260927 02:42:44.268724 8449 ts_tablet_manager.cc:1939] T 739650a58af4408487065aed961fdd36 P 321ba40f052b42d3a1390ef59edb3bce: Deleting consensus metadata
I20260927 02:42:44.268956 7897 catalog_manager.cc:4841] TS 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1:46107): tablet 739650a58af4408487065aed961fdd36 (table test-table [id=1e525bc239ef4cf0809dab5ecc135835]) successfully deleted
I20260927 02:42:44.268970 7899 catalog_manager.cc:4841] TS 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057): tablet 739650a58af4408487065aed961fdd36 (table test-table [id=1e525bc239ef4cf0809dab5ecc135835]) successfully deleted
I20260927 02:42:44.269251 7898 catalog_manager.cc:4841] TS f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227): tablet 739650a58af4408487065aed961fdd36 (table test-table [id=1e525bc239ef4cf0809dab5ecc135835]) successfully deleted
Sep 27 02:42:44 dist-test-slave-44kr krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:42:44 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:44 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:42:44.295761 8464 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:44.290222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.0.98.62:43353)
0927 02:42:44.290290 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.290292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.290382 (+ 90us) server_negotiation.cc:407] Connection header received
0927 02:42:44.290459 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.290460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.290470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.290519 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:44.291100 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.291666 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.292684 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.292894 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.292953 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.295683 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54164: BlockingRecv error: recv got EOF from 127.0.0.1:54164 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 02:42:44 dist-test-slave-44kr krb5kdc[7868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:42:44.303478 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.296691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.0.98.62:43353)
0927 02:42:44.296747 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.296748 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.296848 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:42:44.296901 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.296902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.296911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.296971 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:44.297445 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.297927 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.298586 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.298789 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.298841 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.300959 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:44.300960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:44.300961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:44.300968 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:44.302368 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.302798 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.302799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.302800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.302853 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.303018 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.303019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.303019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.303125 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:44.303272 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.303308 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.303383 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:42:44.304946 7913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:44.305100 7913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:44.306339 8050 tablet_service.cc:1467] Processing CreateTablet for tablet 51cca3f341a64819884bba9a72904050 (DEFAULT_TABLE table=test-table [id=c9074118c4e443689b7ed282fc5a0f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:44.306373 8190 tablet_service.cc:1467] Processing CreateTablet for tablet 51cca3f341a64819884bba9a72904050 (DEFAULT_TABLE table=test-table [id=c9074118c4e443689b7ed282fc5a0f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:44.306367 8330 tablet_service.cc:1467] Processing CreateTablet for tablet 51cca3f341a64819884bba9a72904050 (DEFAULT_TABLE table=test-table [id=c9074118c4e443689b7ed282fc5a0f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:44.306490 8190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cca3f341a64819884bba9a72904050. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.306490 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cca3f341a64819884bba9a72904050. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.306488 8050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cca3f341a64819884bba9a72904050. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.307742 8426 tablet_bootstrap.cc:492] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: Bootstrap starting.
I20260927 02:42:44.307797 8425 tablet_bootstrap.cc:492] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: Bootstrap starting.
I20260927 02:42:44.308145 8425 tablet_bootstrap.cc:654] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:44.308146 8426 tablet_bootstrap.cc:654] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:44.308501 8424 tablet_bootstrap.cc:492] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: Bootstrap starting.
I20260927 02:42:44.308756 8425 tablet_bootstrap.cc:492] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: No bootstrap required, opened a new log
I20260927 02:42:44.308801 8425 ts_tablet_manager.cc:1397] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:44.308821 8424 tablet_bootstrap.cc:654] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:44.308964 8425 raft_consensus.cc:348] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.309033 8425 raft_consensus.cc:374] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.309053 8425 raft_consensus.cc:729] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321ba40f052b42d3a1390ef59edb3bce, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.309093 8425 consensus_queue.cc:260] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.309191 8425 ts_tablet_manager.cc:1428] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 02:42:44.309599 8426 tablet_bootstrap.cc:492] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: No bootstrap required, opened a new log
I20260927 02:42:44.309669 8426 ts_tablet_manager.cc:1397] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:42:44.309720 8424 tablet_bootstrap.cc:492] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: No bootstrap required, opened a new log
I20260927 02:42:44.309754 8424 ts_tablet_manager.cc:1397] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:44.309813 8426 raft_consensus.cc:348] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.309847 8424 raft_consensus.cc:348] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.309880 8426 raft_consensus.cc:374] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.309890 8424 raft_consensus.cc:374] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.309902 8424 raft_consensus.cc:729] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d50c57e6ed64cd9a8c159569e28e610, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.309898 8426 raft_consensus.cc:729] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63b020da5194b2e95d5d8403bba5b69, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.309937 8424 consensus_queue.cc:260] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.309940 8426 consensus_queue.cc:260] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.310014 8424 ts_tablet_manager.cc:1428] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.310029 8426 ts_tablet_manager.cc:1428] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.357084 8435 raft_consensus.cc:482] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:44.357268 8435 raft_consensus.cc:504] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.357419 8435 leader_election.cc:288] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227), 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057)
I20260927 02:42:44.357652 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cca3f341a64819884bba9a72904050" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69" is_pre_election: true
I20260927 02:42:44.357658 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cca3f341a64819884bba9a72904050" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321ba40f052b42d3a1390ef59edb3bce" is_pre_election: true
I20260927 02:42:44.357758 8209 raft_consensus.cc:2454] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 0.
I20260927 02:42:44.357767 8350 raft_consensus.cc:2454] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 0.
I20260927 02:42:44.357930 8003 leader_election.cc:302] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321ba40f052b42d3a1390ef59edb3bce, 8d50c57e6ed64cd9a8c159569e28e610; no voters:
I20260927 02:42:44.358106 8435 raft_consensus.cc:2781] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:44.358148 8435 raft_consensus.cc:482] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:44.358166 8435 raft_consensus.cc:3037] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.359091 8435 raft_consensus.cc:504] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.359270 8435 leader_election.cc:288] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 election: Requested vote from peers f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227), 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057)
I20260927 02:42:44.359483 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cca3f341a64819884bba9a72904050" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69"
I20260927 02:42:44.359493 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cca3f341a64819884bba9a72904050" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321ba40f052b42d3a1390ef59edb3bce"
I20260927 02:42:44.359577 8350 raft_consensus.cc:3037] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.359577 8209 raft_consensus.cc:3037] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.360427 8350 raft_consensus.cc:2454] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 1.
I20260927 02:42:44.360601 8003 leader_election.cc:302] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321ba40f052b42d3a1390ef59edb3bce, 8d50c57e6ed64cd9a8c159569e28e610; no voters:
I20260927 02:42:44.360719 8435 raft_consensus.cc:2781] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:44.360834 8435 raft_consensus.cc:686] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 LEADER]: Becoming Leader. State: Replica: 8d50c57e6ed64cd9a8c159569e28e610, State: Running, Role: LEADER
I20260927 02:42:44.360896 8435 consensus_queue.cc:237] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.361073 8209 raft_consensus.cc:2454] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 1.
I20260927 02:42:44.361591 7912 catalog_manager.cc:5495] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8d50c57e6ed64cd9a8c159569e28e610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:44.371833 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.369506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.0.98.1:46107)
0927 02:42:44.369601 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.369603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.369613 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:44.369636 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.369636 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.369650 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.369711 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:44.369853 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.370374 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.371198 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.371388 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.371451 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.371589 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.371623 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.371668 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.371732 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.371754 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 02:42:44.372633 8350 raft_consensus.cc:1261] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Refusing update from remote peer 8d50c57e6ed64cd9a8c159569e28e610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:44.372706 8209 raft_consensus.cc:1261] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Refusing update from remote peer 8d50c57e6ed64cd9a8c159569e28e610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:44.372962 8497 consensus_queue.cc:1035] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:44.373117 8435 consensus_queue.cc:1035] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:44.375762 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:44.375854 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:44.376523 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: Sending DeleteTablet for 3 replicas of tablet 51cca3f341a64819884bba9a72904050
I20260927 02:42:44.376756 8330 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cca3f341a64819884bba9a72904050 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55920
I20260927 02:42:44.376785 8190 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cca3f341a64819884bba9a72904050 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54996
I20260927 02:42:44.376852 8449 tablet_replica.cc:330] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: stopping tablet replica
I20260927 02:42:44.376900 8449 raft_consensus.cc:2229] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:44.376873 8189 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cca3f341a64819884bba9a72904050 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54996
I20260927 02:42:44.376881 8448 tablet_replica.cc:330] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: stopping tablet replica
I20260927 02:42:44.376946 8449 raft_consensus.cc:2258] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:44.376988 8448 raft_consensus.cc:2229] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:44.377041 8448 raft_consensus.cc:2258] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:44.377085 8050 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cca3f341a64819884bba9a72904050 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260927 02:42:44.377169 8450 tablet_replica.cc:330] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: stopping tablet replica
I20260927 02:42:44.377213 8450 raft_consensus.cc:2229] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:44.377262 8450 raft_consensus.cc:2258] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:44.377472 8448 ts_tablet_manager.cc:1905] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:44.377619 8450 ts_tablet_manager.cc:1905] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:44.377691 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:44.377820 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:44.378304 8449 ts_tablet_manager.cc:1905] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:44.379379 8448 ts_tablet_manager.cc:1918] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:44.379453 8448 log.cc:1196] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/51cca3f341a64819884bba9a72904050
I20260927 02:42:44.379680 8448 ts_tablet_manager.cc:1939] T 51cca3f341a64819884bba9a72904050 P f63b020da5194b2e95d5d8403bba5b69: Deleting consensus metadata
I20260927 02:42:44.379879 8330 tablet_service.cc:1467] Processing CreateTablet for tablet 80ec5c39537a4a65a18bfe3ff69decfc (DEFAULT_TABLE table=test-table [id=ddcd6b5075684626aa0b8c4e44aa76b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:44.379956 8450 ts_tablet_manager.cc:1918] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:44.379993 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ec5c39537a4a65a18bfe3ff69decfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.380007 8450 log.cc:1196] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/51cca3f341a64819884bba9a72904050
I20260927 02:42:44.380232 8450 ts_tablet_manager.cc:1939] T 51cca3f341a64819884bba9a72904050 P 8d50c57e6ed64cd9a8c159569e28e610: Deleting consensus metadata
I20260927 02:42:44.381028 8425 tablet_bootstrap.cc:492] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: Bootstrap starting.
I20260927 02:42:44.381184 7897 catalog_manager.cc:4841] TS 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1:46107): tablet 51cca3f341a64819884bba9a72904050 (table test-table [id=c9074118c4e443689b7ed282fc5a0f1a]) successfully deleted
I20260927 02:42:44.381278 8050 tablet_service.cc:1467] Processing CreateTablet for tablet 80ec5c39537a4a65a18bfe3ff69decfc (DEFAULT_TABLE table=test-table [id=ddcd6b5075684626aa0b8c4e44aa76b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:44.381297 7898 catalog_manager.cc:4841] TS f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227): tablet 51cca3f341a64819884bba9a72904050 (table test-table [id=c9074118c4e443689b7ed282fc5a0f1a]) successfully deleted
I20260927 02:42:44.381392 8050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ec5c39537a4a65a18bfe3ff69decfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.381420 8425 tablet_bootstrap.cc:654] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: Neither blocks nor log segments found. Creating new log.
W20260927 02:42:44.381438 7898 catalog_manager.cc:4808] TS f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227): delete failed for tablet 51cca3f341a64819884bba9a72904050 because the tablet was not found. No further retry: Not found: Tablet not found: 51cca3f341a64819884bba9a72904050
I20260927 02:42:44.381776 8189 tablet_service.cc:1467] Processing CreateTablet for tablet 80ec5c39537a4a65a18bfe3ff69decfc (DEFAULT_TABLE table=test-table [id=ddcd6b5075684626aa0b8c4e44aa76b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:44.381879 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ec5c39537a4a65a18bfe3ff69decfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.379896 8449 ts_tablet_manager.cc:1918] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:44.381949 8449 log.cc:1196] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/51cca3f341a64819884bba9a72904050
I20260927 02:42:44.382155 8449 ts_tablet_manager.cc:1939] T 51cca3f341a64819884bba9a72904050 P 321ba40f052b42d3a1390ef59edb3bce: Deleting consensus metadata
I20260927 02:42:44.382478 8424 tablet_bootstrap.cc:492] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610: Bootstrap starting.
I20260927 02:42:44.382786 8424 tablet_bootstrap.cc:654] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:44.383044 7899 catalog_manager.cc:4841] TS 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057): tablet 51cca3f341a64819884bba9a72904050 (table test-table [id=c9074118c4e443689b7ed282fc5a0f1a]) successfully deleted
I20260927 02:42:44.383306 8426 tablet_bootstrap.cc:492] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: Bootstrap starting.
I20260927 02:42:44.383363 8424 tablet_bootstrap.cc:492] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610: No bootstrap required, opened a new log
I20260927 02:42:44.383404 8424 ts_tablet_manager.cc:1397] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:44.383508 8424 raft_consensus.cc:348] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.383590 8424 raft_consensus.cc:374] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.383612 8424 raft_consensus.cc:729] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d50c57e6ed64cd9a8c159569e28e610, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.383656 8424 consensus_queue.cc:260] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.383697 8426 tablet_bootstrap.cc:654] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:44.383721 8425 tablet_bootstrap.cc:492] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: No bootstrap required, opened a new log
I20260927 02:42:44.383744 8424 ts_tablet_manager.cc:1428] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.383761 8425 ts_tablet_manager.cc:1397] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 02:42:44.383864 8425 raft_consensus.cc:348] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.383929 8425 raft_consensus.cc:374] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.383975 8425 raft_consensus.cc:729] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321ba40f052b42d3a1390ef59edb3bce, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.384037 8425 consensus_queue.cc:260] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.384135 8425 ts_tablet_manager.cc:1428] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.384820 8426 tablet_bootstrap.cc:492] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: No bootstrap required, opened a new log
I20260927 02:42:44.384873 8426 ts_tablet_manager.cc:1397] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:42:44.385010 8426 raft_consensus.cc:348] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.385074 8426 raft_consensus.cc:374] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.385092 8426 raft_consensus.cc:729] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63b020da5194b2e95d5d8403bba5b69, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.385136 8426 consensus_queue.cc:260] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.385217 8426 ts_tablet_manager.cc:1428] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.598902 8502 raft_consensus.cc:482] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:44.599020 8502 raft_consensus.cc:504] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.599249 8502 leader_election.cc:288] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227), 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057)
I20260927 02:42:44.599468 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ec5c39537a4a65a18bfe3ff69decfc" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69" is_pre_election: true
I20260927 02:42:44.599534 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ec5c39537a4a65a18bfe3ff69decfc" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321ba40f052b42d3a1390ef59edb3bce" is_pre_election: true
I20260927 02:42:44.599586 8209 raft_consensus.cc:2454] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 0.
I20260927 02:42:44.599609 8350 raft_consensus.cc:2454] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 0.
I20260927 02:42:44.599826 8003 leader_election.cc:302] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321ba40f052b42d3a1390ef59edb3bce, 8d50c57e6ed64cd9a8c159569e28e610; no voters:
I20260927 02:42:44.599941 8502 raft_consensus.cc:2781] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:44.599998 8502 raft_consensus.cc:482] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:44.600020 8502 raft_consensus.cc:3037] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.600778 8502 raft_consensus.cc:504] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.600915 8502 leader_election.cc:288] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 election: Requested vote from peers f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227), 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057)
I20260927 02:42:44.601101 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ec5c39537a4a65a18bfe3ff69decfc" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321ba40f052b42d3a1390ef59edb3bce"
I20260927 02:42:44.601140 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ec5c39537a4a65a18bfe3ff69decfc" candidate_uuid: "8d50c57e6ed64cd9a8c159569e28e610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63b020da5194b2e95d5d8403bba5b69"
I20260927 02:42:44.601181 8350 raft_consensus.cc:3037] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.601231 8209 raft_consensus.cc:3037] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.601876 8350 raft_consensus.cc:2454] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 1.
I20260927 02:42:44.601962 8209 raft_consensus.cc:2454] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d50c57e6ed64cd9a8c159569e28e610 in term 1.
I20260927 02:42:44.602087 8003 leader_election.cc:302] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321ba40f052b42d3a1390ef59edb3bce, 8d50c57e6ed64cd9a8c159569e28e610; no voters:
I20260927 02:42:44.602180 8502 raft_consensus.cc:2781] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:44.602248 8502 raft_consensus.cc:686] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 LEADER]: Becoming Leader. State: Replica: 8d50c57e6ed64cd9a8c159569e28e610, State: Running, Role: LEADER
I20260927 02:42:44.602303 8502 consensus_queue.cc:237] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } }
I20260927 02:42:44.602878 7913 catalog_manager.cc:5495] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8d50c57e6ed64cd9a8c159569e28e610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d50c57e6ed64cd9a8c159569e28e610" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:44.619560 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.617368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49651 (local address 127.0.98.3:43057)
0927 02:42:44.617411 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.617413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.617524 (+ 111us) server_negotiation.cc:407] Connection header received
0927 02:42:44.617580 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.617580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.617590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.617637 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:44.617757 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.618222 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.618947 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.619134 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.619186 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.619341 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.619373 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.619413 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.619460 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.619481 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:42:44.619577 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.617298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43057 (local address 127.0.98.1:49651)
0927 02:42:44.617461 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:44.617472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:44.617531 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:44.617643 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:44.617644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:44.617645 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:44.617699 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.617701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.618288 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.618289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.618873 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.618875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.619192 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.619194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.619260 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.619275 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.619418 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.619420 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:42:44.619458 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 02:42:44.619506 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260927 02:42:44.621569 8209 raft_consensus.cc:1261] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Refusing update from remote peer 8d50c57e6ed64cd9a8c159569e28e610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:44.621587 8350 raft_consensus.cc:1261] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Refusing update from remote peer 8d50c57e6ed64cd9a8c159569e28e610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:44.621773 8435 consensus_queue.cc:1035] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "321ba40f052b42d3a1390ef59edb3bce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:44.621871 8502 consensus_queue.cc:1035] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f63b020da5194b2e95d5d8403bba5b69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:44.628985 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.626782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44903 (local address 127.0.98.1:46107)
0927 02:42:44.626820 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.626821 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.626828 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:42:44.626850 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.626850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.626859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.626903 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:44.627031 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.627534 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.628315 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.628599 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.628669 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.628809 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.628836 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.628863 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.628907 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.628922 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:42:44.628994 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.626632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.98.3:44903)
0927 02:42:44.626669 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:44.626681 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:44.626718 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:44.626923 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:44.626924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:44.626925 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:42:44.626971 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.626973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.627639 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.627640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.628213 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.628215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.628689 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.628690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.628738 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.628744 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:42:44.628869 (+ 125us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:42:44.628870 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:42:44.628897 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 02:42:44.628936 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:42:44.635726 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:44.635833 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:44.636516 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 431662ff84e6402f97495d3dc2a5aedf: Sending DeleteTablet for 3 replicas of tablet 80ec5c39537a4a65a18bfe3ff69decfc
I20260927 02:42:44.636814 8330 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ec5c39537a4a65a18bfe3ff69decfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55920
I20260927 02:42:44.636853 8189 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ec5c39537a4a65a18bfe3ff69decfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54996
I20260927 02:42:44.636933 8449 tablet_replica.cc:330] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: stopping tablet replica
I20260927 02:42:44.636967 8448 tablet_replica.cc:330] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: stopping tablet replica
I20260927 02:42:44.636996 8449 raft_consensus.cc:2229] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:44.636982 8050 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ec5c39537a4a65a18bfe3ff69decfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260927 02:42:44.637013 8448 raft_consensus.cc:2229] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:44.637037 8449 raft_consensus.cc:2258] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:44.637045 8448 raft_consensus.cc:2258] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:44.637080 8450 tablet_replica.cc:330] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610: stopping tablet replica
I20260927 02:42:44.637137 8450 raft_consensus.cc:2229] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:44.637198 8450 raft_consensus.cc:2258] T 80ec5c39537a4a65a18bfe3ff69decfc P 8d50c57e6ed64cd9a8c159569e28e610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:44.637398 8448 ts_tablet_manager.cc:1905] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:44.637503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 7985
I20260927 02:42:44.637498 8449 ts_tablet_manager.cc:1905] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:44.639254 8449 ts_tablet_manager.cc:1918] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:44.639323 8449 log.cc:1196] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/80ec5c39537a4a65a18bfe3ff69decfc
I20260927 02:42:44.639473 8448 ts_tablet_manager.cc:1918] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:42:44.639514 8449 ts_tablet_manager.cc:1939] T 80ec5c39537a4a65a18bfe3ff69decfc P 321ba40f052b42d3a1390ef59edb3bce: Deleting consensus metadata
I20260927 02:42:44.639528 8448 log.cc:1196] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/80ec5c39537a4a65a18bfe3ff69decfc
I20260927 02:42:44.639704 8448 ts_tablet_manager.cc:1939] T 80ec5c39537a4a65a18bfe3ff69decfc P f63b020da5194b2e95d5d8403bba5b69: Deleting consensus metadata
I20260927 02:42:44.640107 7899 catalog_manager.cc:4841] TS 321ba40f052b42d3a1390ef59edb3bce (127.0.98.3:43057): tablet 80ec5c39537a4a65a18bfe3ff69decfc (table test-table [id=ddcd6b5075684626aa0b8c4e44aa76b2]) successfully deleted
I20260927 02:42:44.640194 7898 catalog_manager.cc:4841] TS f63b020da5194b2e95d5d8403bba5b69 (127.0.98.2:36227): tablet 80ec5c39537a4a65a18bfe3ff69decfc (table test-table [id=ddcd6b5075684626aa0b8c4e44aa76b2]) successfully deleted
W20260927 02:42:44.643733 7897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46107 (error 108)
I20260927 02:42:44.643941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 8125
I20260927 02:42:44.648538 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.644175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.0.1:58304)
0927 02:42:44.648443 (+ 4268us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:44.648495 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4243,"thread_start_us":59,"threads_started":1}
I20260927 02:42:44.648569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 8265
W20260927 02:42:44.648696 7897 catalog_manager.cc:4584] TS 8d50c57e6ed64cd9a8c159569e28e610 (127.0.98.1:46107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80ec5c39537a4a65a18bfe3ff69decfc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46107: connect: Connection refused (error 111)
I20260927 02:42:44.653616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 7881
2026-09-27T02:42:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2998 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:44 dist-test-slave-44kr krb5kdc[8516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:44 dist-test-slave-44kr krb5kdc[8516](info): set up 2 sockets
Sep 27 02:42:44 dist-test-slave-44kr krb5kdc[8516](info): commencing operation
krb5kdc: starting...
W20260927 02:42:45.624744 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:45 dist-test-slave-44kr krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:45.742995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41031
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:45.821806 8529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:45.822031 8529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:45.822059 8529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:45.823269 8529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:45.823318 8529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:45.823335 8529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:45.823350 8529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:45.823365 8529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:45.824958 8529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:45.825217 8529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:45.825429 8529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:45.830430 8535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:45.830446 8534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:45.830566 8529 server_base.cc:1034] running on GCE node
W20260927 02:42:45.830515 8537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:45.830932 8529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:45.831240 8529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:45.832441 8529 hybrid_clock.cc:648] HybridClock initialized: now 1790476965832392 us; error 62 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T02:42:45Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1187 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:45 dist-test-slave-44kr krb5kdc[8542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:45 dist-test-slave-44kr krb5kdc[8542](info): set up 2 sockets
Sep 27 02:42:45 dist-test-slave-44kr krb5kdc[8542](info): commencing operation
krb5kdc: starting...
W20260927 02:42:46.796667 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:46 dist-test-slave-44kr krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:46.910851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:46.990119 8555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:46.990322 8555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:46.990341 8555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:46.991456 8555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:46.991508 8555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:46.991520 8555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:46.991531 8555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:46.991549 8555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:46.993029 8555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:46.993242 8555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:46.993443 8555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:46.998421 8561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:46.998502 8563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:46.998644 8555 server_base.cc:1034] running on GCE node
W20260927 02:42:46.998432 8560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:46.998993 8555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:46.999332 8555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:47.000564 8555 hybrid_clock.cc:648] HybridClock initialized: now 1790476967000549 us; error 31 us; skew 500 ppm
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:47.009287 8555 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:47.009766 8555 webserver.cc:460] Webserver started at http://127.0.98.62:38753/ using document root <none> and password file <none>
I20260927 02:42:47.009969 8555 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:47.010017 8555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:47.010142 8555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:47.011209 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "85ce06d6fb5c4a73a244214637cf7529"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "e0e285b85c91233a8d687465eaaa8d12"
server_key_iv: "5b448cd37d96a4f867c6a1d012c5ca0e"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.011539 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "85ce06d6fb5c4a73a244214637cf7529"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "e0e285b85c91233a8d687465eaaa8d12"
server_key_iv: "5b448cd37d96a4f867c6a1d012c5ca0e"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.012821 8555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:42:47.013458 8570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.013608 8555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:42:47.013689 8555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "85ce06d6fb5c4a73a244214637cf7529"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "e0e285b85c91233a8d687465eaaa8d12"
server_key_iv: "5b448cd37d96a4f867c6a1d012c5ca0e"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.013751 8555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:47.030834 8555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:47.032898 8555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:47.033038 8555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:47.036772 8555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45297
I20260927 02:42:47.036821 8622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45297 every 8 connection(s)
I20260927 02:42:47.037298 8555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:47.037974 8623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:47.040380 8623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: Bootstrap starting.
I20260927 02:42:47.040987 8623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:47.041256 8623 log.cc:824] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:47.041944 8623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: No bootstrap required, opened a new log
I20260927 02:42:47.043560 8623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260927 02:42:47.043707 8623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:47.043730 8623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ce06d6fb5c4a73a244214637cf7529, State: Initialized, Role: FOLLOWER
I20260927 02:42:47.043864 8623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260927 02:42:47.043933 8623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:47.043955 8623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:47.044000 8623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:47.044905 8623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260927 02:42:47.044996 8623 leader_election.cc:302] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85ce06d6fb5c4a73a244214637cf7529; no voters:
I20260927 02:42:47.045166 8623 leader_election.cc:288] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:47.045248 8626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:47.045392 8623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:47.045630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 8555
I20260927 02:42:47.045714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:47.045840 392 external_mini_cluster.cc:1428] Setting key cac8af9276bb0910a7425e4fc080a738
I20260927 02:42:47.045423 8626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 1 LEADER]: Becoming Leader. State: Replica: 85ce06d6fb5c4a73a244214637cf7529, State: Running, Role: LEADER
I20260927 02:42:47.046089 8626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260927 02:42:47.046792 8628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85ce06d6fb5c4a73a244214637cf7529. Latest consensus state: current_term: 1 leader_uuid: "85ce06d6fb5c4a73a244214637cf7529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } } }
I20260927 02:42:47.046792 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85ce06d6fb5c4a73a244214637cf7529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } } }
I20260927 02:42:47.046969 8628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:47.046984 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: This master's current role is: LEADER
W20260927 02:42:47.047916 8644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:47.047969 8644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:47.048590 8640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:47.048774 8640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:47.050293 8640 catalog_manager.cc:1347] Generated new cluster ID: d6175c23870b4eed926f7ee2529940bc
I20260927 02:42:47.050341 8640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:47.057019 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.046288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.0.98.62:45297)
0927 02:42:47.046457 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:47.046467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:47.046667 (+ 200us) server_negotiation.cc:407] Connection header received
0927 02:42:47.047134 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:47.047143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:47.047214 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:47.047378 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:47.048090 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.049366 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.050165 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.050768 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.050865 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.053531 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:47.053536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:47.053536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:47.053549 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:47.055439 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.055794 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.055797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.055797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.055908 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.056082 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.056082 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.056083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.056420 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:47.056584 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:47.056652 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:42:47.056797 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260927 02:42:47.064276 8640 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:47.065017 8640 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:47.075165 8640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: Generated new TSK 0
I20260927 02:42:47.075490 8640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:47.119933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:47.202849 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:47.203079 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:47.203105 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:47.204347 8649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:47.204411 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:47.204450 8649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:47.206033 8649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:47.206271 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:47.206503 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:47.211493 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.211481 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.211483 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:47.211901 8649 server_base.cc:1034] running on GCE node
I20260927 02:42:47.212142 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:47.212373 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:47.213544 8649 hybrid_clock.cc:648] HybridClock initialized: now 1790476967213532 us; error 30 us; skew 500 ppm
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:47.216130 8649 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:47.216674 8649 webserver.cc:460] Webserver started at http://127.0.98.1:44645/ using document root <none> and password file <none>
I20260927 02:42:47.216882 8649 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:47.216931 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:47.217051 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:47.218142 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "24850e5f7a5b42218d89fb8470767cfc"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "d453d83308b6d26da0286c9324f6c0af"
server_key_iv: "e1c052fe890886517a9e80d1e877ae28"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.218473 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "24850e5f7a5b42218d89fb8470767cfc"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "d453d83308b6d26da0286c9324f6c0af"
server_key_iv: "e1c052fe890886517a9e80d1e877ae28"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.219770 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:42:47.220557 8664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.220711 8649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:47.220801 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "24850e5f7a5b42218d89fb8470767cfc"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "d453d83308b6d26da0286c9324f6c0af"
server_key_iv: "e1c052fe890886517a9e80d1e877ae28"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.220866 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:47.234916 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:47.236955 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:47.237104 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:47.237339 8649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:47.237650 8649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:47.237684 8649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.237712 8649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:47.237730 8649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.243252 8649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46103
I20260927 02:42:47.243328 8777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46103 every 8 connection(s)
I20260927 02:42:47.243636 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:47.244797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 8649
I20260927 02:42:47.244899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:47.245090 392 external_mini_cluster.cc:1428] Setting key fe79f219229cf8478a0246b90edcea85
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:42:47.255033 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.244498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58643 (local address 127.0.98.62:45297)
0927 02:42:47.244577 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:47.244579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:47.244809 (+ 230us) server_negotiation.cc:407] Connection header received
0927 02:42:47.246102 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:47.246104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:47.246120 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:47.246175 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:47.247053 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.247680 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.249199 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.249474 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.249551 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.251872 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:47.251874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:47.251874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:47.251882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:47.253300 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.253807 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.253809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.253809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.253875 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.254268 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.254269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.254270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.254395 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:47.254458 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:47.254858 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 02:42:47.254940 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:42:47.255231 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.244368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:58643)
0927 02:42:47.244616 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:47.244683 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:47.245948 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:47.246294 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:47.246299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:47.246674 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:47.246972 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.246980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.247763 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.247764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.249122 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.249125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.249555 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.249555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.249676 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.250213 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:47.250223 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:47.251762 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:47.253386 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:47.253394 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:47.253397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:47.253714 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:47.253949 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:47.253949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:47.253950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:47.254096 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:47.254472 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:47.254479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:47.254664 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 02:42:47.254808 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 02:42:47.254982 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260927 02:42:47.255934 8778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45297
I20260927 02:42:47.256078 8778 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:47.256321 8778 heartbeater.cc:511] Master 127.0.98.62:45297 requested a full tablet report, sending...
I20260927 02:42:47.256979 8587 ts_manager.cc:194] Registered new tserver with Master: 24850e5f7a5b42218d89fb8470767cfc (127.0.98.1:46103)
I20260927 02:42:47.257695 8587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:47.273327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:47.352805 8783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:47.353008 8783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:47.353029 8783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:47.354115 8783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:47.354148 8783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:47.354171 8783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:47.355619 8783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:47.355805 8783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:47.355981 8783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:47.360697 8788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.360713 8791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.360775 8789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:47.360870 8783 server_base.cc:1034] running on GCE node
I20260927 02:42:47.361191 8783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:47.361425 8783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:47.362601 8783 hybrid_clock.cc:648] HybridClock initialized: now 1790476967362588 us; error 26 us; skew 500 ppm
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:47.365168 8783 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:47.365654 8783 webserver.cc:460] Webserver started at http://127.0.98.2:43215/ using document root <none> and password file <none>
I20260927 02:42:47.365850 8783 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:47.365898 8783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:47.366022 8783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:47.367035 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "8065274a786d4746bf45f37c5a0b720f"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "75f3109dcefb54627a8281ee85390969"
server_key_iv: "55c2aaca369181d0ba76fc66e516b234"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.367372 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8065274a786d4746bf45f37c5a0b720f"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "75f3109dcefb54627a8281ee85390969"
server_key_iv: "55c2aaca369181d0ba76fc66e516b234"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.368676 8783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:42:47.369326 8798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.369481 8783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:42:47.369560 8783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "8065274a786d4746bf45f37c5a0b720f"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "75f3109dcefb54627a8281ee85390969"
server_key_iv: "55c2aaca369181d0ba76fc66e516b234"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.369616 8783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:47.400456 8783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:47.402690 8783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:47.402832 8783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:47.403070 8783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:47.403367 8783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:47.403400 8783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.403429 8783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:47.403455 8783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.409132 8783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45855
I20260927 02:42:47.409199 8911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45855 every 8 connection(s)
I20260927 02:42:47.409503 8783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:47.418483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 8783
I20260927 02:42:47.418606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:47.418810 392 external_mini_cluster.cc:1428] Setting key 5fd93ab7e4d17e4850a8abc4af132343
I20260927 02:42:47.418984 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.410195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53759 (local address 127.0.98.62:45297)
0927 02:42:47.410241 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:47.410243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:47.410524 (+ 281us) server_negotiation.cc:407] Connection header received
0927 02:42:47.411396 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:47.411398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:47.411408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:47.411480 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:47.412067 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.412620 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.413637 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.413828 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.413882 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.416239 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:47.416241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:47.416241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:47.416248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:47.417702 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.418049 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.418050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.418050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.418103 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.418310 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.418310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.418311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.418409 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:47.418456 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:47.418823 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 02:42:47.418894 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:42:47.419252 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.410188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:53759)
0927 02:42:47.410349 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:47.410403 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:47.411297 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:47.411535 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:47.411539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:47.411792 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:47.411984 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.411989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.412718 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.412719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.413556 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.413558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.413894 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.413895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.413994 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.414532 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:47.414540 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:47.416071 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:47.417773 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:47.417778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:47.417779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:47.417990 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:47.418148 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:47.418149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:47.418149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:47.418259 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:47.418463 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:47.418469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:47.418639 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 02:42:47.418804 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 02:42:47.418972 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260927 02:42:47.419747 8912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45297
I20260927 02:42:47.419833 8912 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:47.419999 8912 heartbeater.cc:511] Master 127.0.98.62:45297 requested a full tablet report, sending...
I20260927 02:42:47.420326 8587 ts_manager.cc:194] Registered new tserver with Master: 8065274a786d4746bf45f37c5a0b720f (127.0.98.2:45855)
I20260927 02:42:47.420717 8587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53759
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:47.444209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:47.522924 8917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:47.523141 8917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:47.523175 8917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:47.524497 8917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:47.524541 8917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:47.524578 8917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:47.526170 8917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:47.526427 8917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:47.526646 8917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:47.531472 8923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.531463 8922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.531466 8925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:47.532295 8917 server_base.cc:1034] running on GCE node
I20260927 02:42:47.532585 8917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:47.532845 8917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:47.534071 8917 hybrid_clock.cc:648] HybridClock initialized: now 1790476967534044 us; error 37 us; skew 500 ppm
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:47.536671 8917 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:47.537140 8917 webserver.cc:460] Webserver started at http://127.0.98.3:42733/ using document root <none> and password file <none>
I20260927 02:42:47.537389 8917 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:47.537441 8917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:47.537572 8917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:47.538637 8917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "3fa91470c29d4224b1fcdbacc8dc90dc"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "6e60c3216dce6ad4a68ceb12efed5deb"
server_key_iv: "1c45eb761eddab9ab180f66141af1e6d"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.538981 8917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3fa91470c29d4224b1fcdbacc8dc90dc"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "6e60c3216dce6ad4a68ceb12efed5deb"
server_key_iv: "1c45eb761eddab9ab180f66141af1e6d"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.540329 8917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:42:47.541121 8932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.541280 8917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:47.541359 8917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "3fa91470c29d4224b1fcdbacc8dc90dc"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "6e60c3216dce6ad4a68ceb12efed5deb"
server_key_iv: "1c45eb761eddab9ab180f66141af1e6d"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.541420 8917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:47.550570 8917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:47.552784 8917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:47.552922 8917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:47.553138 8917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:47.553423 8917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:47.553457 8917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.553484 8917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:47.553507 8917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.559135 8917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37917
I20260927 02:42:47.559165 9045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37917 every 8 connection(s)
I20260927 02:42:47.559512 8917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:47.569169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 8917
I20260927 02:42:47.569212 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.560292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59531 (local address 127.0.98.62:45297)
0927 02:42:47.560364 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:47.560367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:47.560575 (+ 208us) server_negotiation.cc:407] Connection header received
0927 02:42:47.561582 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:47.561584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:47.561596 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:47.561654 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:47.562320 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.562855 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.563933 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.564197 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.564257 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.566485 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:47.566486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:47.566487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:47.566495 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:47.567619 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.568101 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.568102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.568102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.568156 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.568448 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.568448 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.568449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.568557 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:47.568610 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:47.568968 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 02:42:47.569111 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:42:47.569247 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.560255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:59531)
0927 02:42:47.560450 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:47.560491 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:47.561455 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:47.561747 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:47.561750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:47.562013 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:47.562222 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.562227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.562959 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.562960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.563844 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.563846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.564295 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.564296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.564469 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.564975 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:47.564984 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:47.566398 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:47.567743 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:47.567750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:47.567752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:47.567996 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:47.568238 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:47.568239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:47.568240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:47.568349 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:47.568642 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:47.568649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:47.568789 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 02:42:47.568880 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:42:47.569005 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260927 02:42:47.569279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:47.569468 392 external_mini_cluster.cc:1428] Setting key 444ae90b47e440fe8ca6c138c5c777c1
I20260927 02:42:47.569846 9046 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45297
I20260927 02:42:47.569955 9046 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:47.570154 9046 heartbeater.cc:511] Master 127.0.98.62:45297 requested a full tablet report, sending...
I20260927 02:42:47.570513 8587 ts_manager.cc:194] Registered new tserver with Master: 3fa91470c29d4224b1fcdbacc8dc90dc (127.0.98.3:37917)
I20260927 02:42:47.570904 8587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59531
I20260927 02:42:47.581126 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:42:47.647360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 8649
I20260927 02:42:47.651536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 8783
I20260927 02:42:47.655422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 8917
I20260927 02:42:47.658998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 8555
I20260927 02:42:47.663230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45297
--webserver_interface=127.0.98.62
--webserver_port=38753
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:47.743947 9055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:47.744160 9055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:47.744182 9055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:47.745402 9055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:47.745447 9055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:47.745461 9055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:47.745472 9055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:47.745483 9055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:47.747007 9055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:47.747257 9055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:47.747473 9055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:47.752185 9061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.752288 9063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.752254 9060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:47.752545 9055 server_base.cc:1034] running on GCE node
I20260927 02:42:47.752885 9055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:47.753165 9055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:47.754307 9055 hybrid_clock.cc:648] HybridClock initialized: now 1790476967754284 us; error 34 us; skew 500 ppm
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:47.756695 9055 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:47.757121 9055 webserver.cc:460] Webserver started at http://127.0.98.62:38753/ using document root <none> and password file <none>
I20260927 02:42:47.757323 9055 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:47.757382 9055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:47.758821 9055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:47.759552 9070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.759773 9055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:47.759860 9055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "85ce06d6fb5c4a73a244214637cf7529"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "e0e285b85c91233a8d687465eaaa8d12"
server_key_iv: "5b448cd37d96a4f867c6a1d012c5ca0e"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.760244 9055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:47.779076 9055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:47.781284 9055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:47.781443 9055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:47.785120 9055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45297
I20260927 02:42:47.785159 9122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45297 every 8 connection(s)
I20260927 02:42:47.785660 9055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:47.786909 9123 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:42:47.787564 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: Bootstrap starting.
I20260927 02:42:47.787909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 9055
I20260927 02:42:47.788184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46103
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45297
--webserver_port=44645
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:42:47.795697 9123 log.cc:824] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:47.797652 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:42:47.797931 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: Bootstrap complete.
I20260927 02:42:47.800031 9123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260927 02:42:47.800288 9123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ce06d6fb5c4a73a244214637cf7529, State: Initialized, Role: FOLLOWER
I20260927 02:42:47.800491 9123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260927 02:42:47.800565 9123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:47.800606 9123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:47.800657 9123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:42:47.801781 9123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260927 02:42:47.801877 9123 leader_election.cc:302] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85ce06d6fb5c4a73a244214637cf7529; no voters:
I20260927 02:42:47.802067 9123 leader_election.cc:288] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:42:47.802134 9127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:42:47.802318 9127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [term 2 LEADER]: Becoming Leader. State: Replica: 85ce06d6fb5c4a73a244214637cf7529, State: Running, Role: LEADER
I20260927 02:42:47.802357 9123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:47.802470 9127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260927 02:42:47.802767 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85ce06d6fb5c4a73a244214637cf7529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } } }
I20260927 02:42:47.802843 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:47.803061 9129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85ce06d6fb5c4a73a244214637cf7529. Latest consensus state: current_term: 2 leader_uuid: "85ce06d6fb5c4a73a244214637cf7529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ce06d6fb5c4a73a244214637cf7529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } } }
I20260927 02:42:47.803164 9129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:47.803285 9133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:47.803788 9133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:47.803908 9133 catalog_manager.cc:1259] Loaded cluster ID: d6175c23870b4eed926f7ee2529940bc
I20260927 02:42:47.803946 9133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:47.804849 9133 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:47.804994 9133 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85ce06d6fb5c4a73a244214637cf7529: Loaded TSK: 0
I20260927 02:42:47.805152 9133 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:42:47.872318 9125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:47.872586 9125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:47.872622 9125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:47.873803 9125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:47.873847 9125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:47.873872 9125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:47.875303 9125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44645
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:47.875510 9125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:47.875731 9125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:47.881762 9149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.881804 9152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:47.882141 9125 server_base.cc:1034] running on GCE node
W20260927 02:42:47.881870 9150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:47.882453 9125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:47.882710 9125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:47.883872 9125 hybrid_clock.cc:648] HybridClock initialized: now 1790476967883856 us; error 30 us; skew 500 ppm
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:47.886596 9125 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:42:47.887090 9125 webserver.cc:460] Webserver started at http://127.0.98.1:44645/ using document root <none> and password file <none>
I20260927 02:42:47.887302 9125 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:47.887362 9125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:47.888790 9125 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:47.889453 9159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.889662 9125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:42:47.889757 9125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "24850e5f7a5b42218d89fb8470767cfc"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "d453d83308b6d26da0286c9324f6c0af"
server_key_iv: "e1c052fe890886517a9e80d1e877ae28"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.890118 9125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:47.900599 9125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:47.902937 9125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:47.903139 9125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:47.903373 9125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:47.903708 9125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:47.903751 9125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.903788 9125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:47.903805 9125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.910387 9125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46103
I20260927 02:42:47.910431 9272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46103 every 8 connection(s)
I20260927 02:42:47.910755 9125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:47.913219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 9125
I20260927 02:42:47.913491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45855
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45297
--webserver_port=43215
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:47.919885 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.911501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:37963)
0927 02:42:47.911684 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:47.911730 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:47.912333 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:47.912860 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:47.912864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:47.913639 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:47.913913 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.913920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.915360 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.915361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.916204 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.916206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.916855 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.916856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.916984 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.917494 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:47.917503 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:47.919650 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
W20260927 02:42:47.920013 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:47.911618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37963 (local address 127.0.98.62:45297)
0927 02:42:47.911769 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:47.911778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:47.911815 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:42:47.912572 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:47.912627 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:47.912675 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:47.912787 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:47.914046 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.915259 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.916294 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.916759 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.916868 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.919829 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37963: BlockingRecv error: recv got EOF from 127.0.98.1:37963 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
W20260927 02:42:47.920495 9273 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45297 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45297: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:47.924881 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.920717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:59783)
0927 02:42:47.920760 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:47.920771 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:47.920813 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:47.920980 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:47.920981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:47.921124 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:47.921176 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.921178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.921829 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.921830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.922404 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.922407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.922744 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.922745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.922807 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.923193 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:47.923198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:47.924811 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:42:47.924973 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:47.920776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59783 (local address 127.0.98.62:45297)
0927 02:42:47.920878 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:47.920880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:47.920887 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:42:47.920911 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:47.920912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:47.920922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:47.920969 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:47.921247 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.921757 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.922479 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.922683 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.922739 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.924912 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59783: BlockingRecv error: recv got EOF from 127.0.98.1:59783 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:47 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:47.929354 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.925353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:38451)
0927 02:42:47.925436 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:47.925446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:47.925488 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:47.925603 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:47.925603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:47.925724 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:47.925753 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.925754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.926347 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.926348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.926847 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:47.926849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.927207 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.927208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:47.927262 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.927635 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:47.927640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:47.929275 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 02:42:47.929412 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:47.925401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38451 (local address 127.0.98.62:45297)
0927 02:42:47.925436 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:47.925438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:47.925475 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:42:47.925534 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:47.925535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:47.925545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:47.925591 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:47.925815 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.926280 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.926917 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.927105 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.927162 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.929361 (+ 2199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38451: BlockingRecv error: recv got EOF from 127.0.98.1:38451 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:42:47.929718 9273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:42:48.001053 9277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:48.001343 9277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:48.001384 9277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:48.003078 9277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:48.003137 9277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:48.003181 9277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:48.004923 9277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43215
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:48.005149 9277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:48.005343 9277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:48.010430 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:48.010430 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:48.010550 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:48.010465 9277 server_base.cc:1034] running on GCE node
I20260927 02:42:48.010811 9277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:48.011067 9277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:48.012253 9277 hybrid_clock.cc:648] HybridClock initialized: now 1790476968012237 us; error 34 us; skew 500 ppm
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:48.014709 9277 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:42:48.015220 9277 webserver.cc:460] Webserver started at http://127.0.98.2:43215/ using document root <none> and password file <none>
I20260927 02:42:48.015497 9277 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:48.015623 9277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:48.017076 9277 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:48.017688 9292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:48.017856 9277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:48.017940 9277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "8065274a786d4746bf45f37c5a0b720f"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "75f3109dcefb54627a8281ee85390969"
server_key_iv: "55c2aaca369181d0ba76fc66e516b234"
server_key_version: "encryptionkey@0"
I20260927 02:42:48.018288 9277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:48.029263 9277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:48.031306 9277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:48.031451 9277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:48.031677 9277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:48.031972 9277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:48.032009 9277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:48.032042 9277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:48.032061 9277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:48.038367 9277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45855
I20260927 02:42:48.038408 9405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45855 every 8 connection(s)
I20260927 02:42:48.038790 9277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:48.038945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 9277
I20260927 02:42:48.039255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37917
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45297
--webserver_port=42733
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:42:48.047145 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:48.039557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46671 (local address 127.0.98.62:45297)
0927 02:42:48.039668 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.039670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.039927 (+ 257us) server_negotiation.cc:407] Connection header received
0927 02:42:48.040499 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.040500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.040527 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.040585 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.041945 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.042458 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.043529 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.043732 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.043794 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.047085 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46671: BlockingRecv error: recv got EOF from 127.0.98.2:46671 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 02:42:48.047199 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.039568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:46671)
0927 02:42:48.039777 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:48.039822 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:48.040359 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:48.040728 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:48.040733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:48.041512 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:48.041841 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.041849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.042563 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.042564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.043438 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.043441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.043810 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.043811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.043948 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.044589 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:48.044598 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:48.046982 (+ 2384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
W20260927 02:42:48.047375 9406 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45297 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45297: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:48.051883 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.047600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:46641)
0927 02:42:48.047645 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:48.047657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:48.047691 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:48.047832 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:48.047833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:48.047951 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:48.047995 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.047997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.048723 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.048724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.049214 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.049216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.049537 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.049538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.049597 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.050075 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:48.050080 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:48.051830 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:42:48.051992 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:48.047664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46641 (local address 127.0.98.62:45297)
0927 02:42:48.047725 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.047726 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.047735 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:48.047760 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.047761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.047770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.047828 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.048129 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.048655 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.049289 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.049469 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.049529 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.051930 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46641: BlockingRecv error: recv got EOF from 127.0.98.2:46641 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:48.056332 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.052115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:60865)
0927 02:42:48.052246 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:48.052255 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:48.052303 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:48.052459 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:48.052460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:48.052566 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:48.052595 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.052596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.053472 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.053473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.053991 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.053992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.054411 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.054412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.054447 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.054853 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:48.054857 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:48.056282 (+ 1425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
W20260927 02:42:48.056460 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:48.052159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60865 (local address 127.0.98.62:45297)
0927 02:42:48.052188 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.052190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.052307 (+ 117us) server_negotiation.cc:407] Connection header received
0927 02:42:48.052337 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.052338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.052347 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.052429 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.052660 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.053389 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.054064 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.054340 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.054404 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.056376 (+ 1972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60865: BlockingRecv error: recv got EOF from 127.0.98.2:60865 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260927 02:42:48.056471 9406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:42:48.122721 9409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:48.122929 9409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:48.122953 9409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:48.124190 9409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:48.124230 9409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:48.124256 9409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:48.125799 9409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42733
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:42:48.126040 9409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:48.126272 9409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:48.131086 9415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:48.131091 9418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:48.131174 9416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:48.131351 9409 server_base.cc:1034] running on GCE node
I20260927 02:42:48.131620 9409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:48.131865 9409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:48.133011 9409 hybrid_clock.cc:648] HybridClock initialized: now 1790476968132994 us; error 29 us; skew 500 ppm
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:48.135532 9409 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:42:48.135963 9409 webserver.cc:460] Webserver started at http://127.0.98.3:42733/ using document root <none> and password file <none>
I20260927 02:42:48.136179 9409 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:48.136241 9409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:48.137672 9409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:48.138382 9425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:48.138581 9409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:42:48.138670 9409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "3fa91470c29d4224b1fcdbacc8dc90dc"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-44kr"
server_key: "6e60c3216dce6ad4a68ceb12efed5deb"
server_key_iv: "1c45eb761eddab9ab180f66141af1e6d"
server_key_version: "encryptionkey@0"
I20260927 02:42:48.139079 9409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:48.151194 9409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:48.153420 9409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:48.153566 9409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:48.153793 9409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:48.154094 9409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:48.154126 9409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:48.154167 9409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:48.154191 9409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:48.159806 9409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37917
I20260927 02:42:48.159866 9538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37917 every 8 connection(s)
I20260927 02:42:48.160234 9409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:48.165026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 9409
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:48.167905 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.160975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:56315)
0927 02:42:48.161110 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:48.161146 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:48.162169 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:48.162457 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:48.162461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:48.162716 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:48.162931 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.162936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.163635 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.163636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.164557 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.164559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.164979 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.164979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.165073 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.165680 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:48.165688 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:48.167752 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
W20260927 02:42:48.167912 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:48.160987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56315 (local address 127.0.98.62:45297)
0927 02:42:48.161035 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.161037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.161278 (+ 241us) server_negotiation.cc:407] Connection header received
0927 02:42:48.162309 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.162311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.162322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.162377 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.163003 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.163540 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.164646 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.164886 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.164982 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.167849 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56315: BlockingRecv error: recv got EOF from 127.0.98.3:56315 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 02:42:48.168138 9539 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45297 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45297: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:48.172668 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.165604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.0.98.62:45297)
0927 02:42:48.165715 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.165717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.165825 (+ 108us) server_negotiation.cc:407] Connection header received
0927 02:42:48.165893 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.165893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.165905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.165966 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.166623 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.167187 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.168328 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.168594 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.168658 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.169783 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:48.169788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:48.169788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:48.169801 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:48.171395 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:48.171709 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:48.171720 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:48.171721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:48.171830 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:48.171995 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:48.171996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:48.171996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:48.172220 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:48.172341 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:48.172470 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 02:42:48.172571 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260927 02:42:48.172683 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.168376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:33681)
0927 02:42:48.168431 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:48.168442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:48.168479 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:48.168597 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:48.168598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:48.168746 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:48.168791 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.168792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.169383 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.169384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.169878 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.169880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.170348 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.170349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.170406 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.170852 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:48.170856 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:48.172612 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 02:42:48.172830 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:48.168440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33681 (local address 127.0.98.62:45297)
0927 02:42:48.168496 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.168498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.168516 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:42:48.168535 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.168536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.168545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.168594 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.168866 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.169317 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.169948 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.170278 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.170335 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.172775 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33681: BlockingRecv error: recv got EOF from 127.0.98.3:33681 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:48.176847 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.172950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:60793)
0927 02:42:48.172993 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:48.173003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:48.173038 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:48.173316 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:48.173317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:48.173448 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:48.173476 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.173477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.174127 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.174127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.174594 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.174595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.174915 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.174916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.174949 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.175313 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:48.175317 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:48.176803 (+ 1486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:42:48.176935 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:48.172987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60793 (local address 127.0.98.62:45297)
0927 02:42:48.173161 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.173163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.173170 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:42:48.173187 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.173188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.173198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.173256 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.173536 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.174059 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.174670 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.174857 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.174914 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.176880 (+ 1966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60793: BlockingRecv error: recv got EOF from 127.0.98.3:60793 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
W20260927 02:42:48.176976 9539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:48 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:48.935216 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.930149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:43869)
0927 02:42:48.930365 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:48.930378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:48.930449 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:48.930617 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:48.930617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:48.930770 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:48.930816 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.930817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.931562 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.931564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.932137 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:48.932140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.932597 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.932598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:48.932652 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.933185 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:48.933190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:48.935127 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":130,"threads_started":1}
W20260927 02:42:48.935288 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:48.930243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43869 (local address 127.0.98.62:45297)
0927 02:42:48.930388 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.930390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.930445 (+ 55us) server_negotiation.cc:407] Connection header received
0927 02:42:48.930493 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.930494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.930503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.930566 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.930890 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.931473 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.932228 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.932473 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.932528 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.935229 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43869: BlockingRecv error: recv got EOF from 127.0.98.1:43869 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
Sep 27 02:42:49 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:49 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:49.061733 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:49.056884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:55765)
0927 02:42:49.057020 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:49.057030 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:49.057088 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:49.057199 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:49.057199 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:49.057352 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:49.057399 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:49.057400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.058085 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.058086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:49.058660 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:49.058662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.059108 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.059109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:49.059171 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:49.059716 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:49.059721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:49.061655 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
W20260927 02:42:49.061847 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:49.056969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55765 (local address 127.0.98.62:45297)
0927 02:42:49.057014 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:49.057016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:49.057073 (+ 57us) server_negotiation.cc:407] Connection header received
0927 02:42:49.057128 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:49.057129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:49.057138 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:49.057189 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:49.057454 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.057957 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.058766 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.058957 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.059014 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:49.061785 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55765: BlockingRecv error: recv got EOF from 127.0.98.2:55765 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 02:42:49 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:49 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:49.182219 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:49.177369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:46927)
0927 02:42:49.177524 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:49.177536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:49.177605 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:49.177793 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:49.177794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:49.177949 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:49.177995 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:49.177996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.178691 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.178692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:49.179305 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:49.179308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.179688 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.179689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:49.179743 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:49.180232 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:49.180236 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:49.182144 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
W20260927 02:42:49.182312 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:49.177485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46927 (local address 127.0.98.62:45297)
0927 02:42:49.177543 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:49.177544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:49.177655 (+ 111us) server_negotiation.cc:407] Connection header received
0927 02:42:49.177681 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:49.177682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:49.177695 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:49.177742 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:49.178073 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.178602 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.179394 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.179590 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.179649 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:49.182242 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46927: BlockingRecv error: recv got EOF from 127.0.98.3:46927 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 02:42:49 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:49 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:49.940676 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:49.935782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:58695)
0927 02:42:49.935937 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:49.935949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:49.936012 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:49.936142 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:49.936142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:49.936297 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:49.936344 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:49.936346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.937037 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.937038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:49.937731 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:49.937735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.938128 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.938129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:49.938180 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:49.938661 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:49.938666 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:49.940597 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
W20260927 02:42:49.940768 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:49.935906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58695 (local address 127.0.98.62:45297)
0927 02:42:49.936016 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:49.936018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:49.936028 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:49.936054 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:49.936055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:49.936064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:49.936127 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:49.936429 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.936952 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.937835 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:49.938037 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:49.938094 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:49.940696 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58695: BlockingRecv error: recv got EOF from 127.0.98.1:58695 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
Sep 27 02:42:50 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:50 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:50.066817 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:50.062258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:44361)
0927 02:42:50.062397 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:50.062408 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:50.062480 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:50.062630 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:50.062631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:50.062802 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:50.062850 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:50.062852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.063585 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.063586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:50.064115 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:50.064117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.064545 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.064546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:50.064600 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:50.065071 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:50.065076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:50.066759 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
W20260927 02:42:50.066969 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:50.062336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44361 (local address 127.0.98.62:45297)
0927 02:42:50.062388 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:50.062389 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:50.062468 (+ 79us) server_negotiation.cc:407] Connection header received
0927 02:42:50.062521 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:50.062521 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:50.062531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:50.062586 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:50.062927 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.063517 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.064212 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.064459 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.064515 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:50.066899 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44361: BlockingRecv error: recv got EOF from 127.0.98.2:44361 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 02:42:50 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:50 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:50.187352 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:50.182772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:58391)
0927 02:42:50.182931 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:50.182944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:50.183015 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:50.183172 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:50.183173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:50.183385 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:50.183432 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:50.183434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.184082 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.184083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:50.184646 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:50.184648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.185005 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.185005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:50.185060 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:50.185520 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:50.185525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:50.187282 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
W20260927 02:42:50.187489 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:50.182955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58391 (local address 127.0.98.62:45297)
0927 02:42:50.183019 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:50.183021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:50.183029 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:50.183094 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:50.183095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:50.183108 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:50.183155 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:50.183515 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.184002 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.184755 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.184938 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.184995 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:50.187418 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58391: BlockingRecv error: recv got EOF from 127.0.98.3:58391 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 02:42:50 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:50 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:50.945832 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:50.941175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:58239)
0927 02:42:50.941310 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:50.941321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:50.941370 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:50.941612 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:50.941613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:50.941801 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:50.941850 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:50.941852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.942617 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.942618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:50.943152 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:50.943153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.943528 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.943530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:50.943581 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:50.943998 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:50.944002 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:50.945773 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
W20260927 02:42:50.945931 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:50.941300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58239 (local address 127.0.98.62:45297)
0927 02:42:50.941452 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:50.941454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:50.941464 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:50.941487 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:50.941487 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:50.941497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:50.941562 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:50.941943 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.942517 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.943247 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:50.943439 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:50.943493 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:50.945861 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58239: BlockingRecv error: recv got EOF from 127.0.98.1:58239 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
Sep 27 02:42:51 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:51 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:51.071614 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:51.067269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:45697)
0927 02:42:51.067439 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:51.067450 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:51.067509 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:51.067624 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:51.067624 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:51.067785 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:51.067833 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:51.067834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.068544 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.068545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:51.069067 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:51.069068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.069411 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.069412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:51.069448 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:51.069860 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:51.069864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:51.071548 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
W20260927 02:42:51.071697 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:51.067343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45697 (local address 127.0.98.62:45297)
0927 02:42:51.067388 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:51.067390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:51.067505 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:42:51.067562 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:51.067563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:51.067572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:51.067611 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:51.067894 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.068488 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.069167 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.069358 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.069409 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:51.071638 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45697: BlockingRecv error: recv got EOF from 127.0.98.2:45697 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 02:42:51 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:51 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:51.192279 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:51.187796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:49061)
0927 02:42:51.187959 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:51.187970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:51.188031 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:51.188136 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:51.188137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:51.188324 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:51.188372 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:51.188374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.189021 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.189022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:51.189578 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:51.189579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.189890 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.189890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:51.189928 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:51.190360 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:51.190365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:51.192156 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
W20260927 02:42:51.192312 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:51.187884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49061 (local address 127.0.98.62:45297)
0927 02:42:51.187948 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:51.187950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:51.188027 (+ 77us) server_negotiation.cc:407] Connection header received
0927 02:42:51.188072 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:51.188073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:51.188084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:51.188130 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:51.188461 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.188954 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.189645 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.189835 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.189885 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:51.192261 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49061: BlockingRecv error: recv got EOF from 127.0.98.3:49061 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 02:42:51 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:51 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:51.950939 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:51.946355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:49429)
0927 02:42:51.946552 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:51.946566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:51.946637 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:51.946793 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:51.946793 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:51.946958 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:51.947005 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:51.947007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.947712 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.947713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:51.948258 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:51.948260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.948627 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.948628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:51.948672 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:51.949190 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:51.949194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:51.950880 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
W20260927 02:42:51.951081 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:51.946465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49429 (local address 127.0.98.62:45297)
0927 02:42:51.946619 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:51.946621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:51.946632 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:51.946705 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:51.946705 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:51.946715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:51.946778 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:51.947101 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.947609 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.948343 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.948566 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.948620 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:51.951007 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49429: BlockingRecv error: recv got EOF from 127.0.98.1:49429 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
Sep 27 02:42:52 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:52 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:52.079488 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:52.072070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:34067)
0927 02:42:52.072183 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:52.072194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:52.072258 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:52.072413 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:52.072414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:52.072585 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:52.072661 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:52.072663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.073305 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.073306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:52.073829 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:52.073831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.074160 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.074161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:52.074212 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:52.077674 (+ 3462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:52.077679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:52.079410 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
W20260927 02:42:52.079582 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:52.072189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34067 (local address 127.0.98.62:45297)
0927 02:42:52.072278 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:52.072280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:52.072289 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:52.072314 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:52.072315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:52.072327 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:52.072375 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:52.072739 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.073234 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.073901 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.074082 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.074148 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:52.079527 (+ 5379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34067: BlockingRecv error: recv got EOF from 127.0.98.2:34067 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 02:42:52 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:52 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:52.197180 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:52.192784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:36103)
0927 02:42:52.192906 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:52.192916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:52.192969 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:52.193093 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:52.193094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:52.193295 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:52.193345 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:52.193347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.194036 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.194037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:52.194576 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:52.194578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.194941 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.194942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:52.194981 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:52.195418 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:52.195423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:52.197120 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
W20260927 02:42:52.197312 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:52.192860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36103 (local address 127.0.98.62:45297)
0927 02:42:52.192915 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:52.192916 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:52.192958 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:42:52.193009 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:52.193010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:52.193021 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:52.193072 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:52.193441 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.193933 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.194674 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.194879 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.194939 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:52.197240 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36103: BlockingRecv error: recv got EOF from 127.0.98.3:36103 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 02:42:52 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:52 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:52.956377 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:52.951489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:53109)
0927 02:42:52.951637 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:52.951649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:52.951710 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:52.951873 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:52.951873 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:52.952028 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:52.952075 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:52.952076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.952872 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.952873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:52.953557 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:52.953560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.953979 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.953980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:52.954016 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:52.954415 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:52.954418 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:52.956288 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
W20260927 02:42:52.956493 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:52.951624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53109 (local address 127.0.98.62:45297)
0927 02:42:52.951756 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:52.951758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:52.951767 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:52.951793 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:52.951794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:52.951804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:52.951861 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:52.952164 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.952782 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.953656 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.953909 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.953980 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:52.956430 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53109: BlockingRecv error: recv got EOF from 127.0.98.1:53109 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
Sep 27 02:42:53 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:53 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:53.085300 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:53.080200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:47693)
0927 02:42:53.080374 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:53.080410 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:53.080486 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:53.080684 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:53.080685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:53.080852 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:53.080898 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:53.080900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.081569 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.081571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:53.082114 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:53.082116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.082452 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.082453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:53.082508 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:53.082982 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:53.082987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:53.085171 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
W20260927 02:42:53.085327 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:53.080419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47693 (local address 127.0.98.62:45297)
0927 02:42:53.080465 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:53.080468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:53.080478 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:53.080553 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:53.080554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:53.080565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:53.080651 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:53.080977 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.081478 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.082202 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.082389 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.082446 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:53.085276 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47693: BlockingRecv error: recv got EOF from 127.0.98.2:47693 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 02:42:53 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:53 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:53.202740 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:53.197719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:44615)
0927 02:42:53.197913 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:53.197924 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:53.197977 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:53.198149 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:53.198151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:53.198351 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:53.198398 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:53.198400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.199080 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.199081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:53.199629 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:53.199630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.199956 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.199957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:53.199993 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:53.200545 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:53.200550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:53.202676 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
W20260927 02:42:53.202975 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:53.197894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44615 (local address 127.0.98.62:45297)
0927 02:42:53.197984 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:53.197986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:53.197996 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:53.198020 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:53.198021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:53.198032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:53.198097 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:53.198502 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.199007 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.199711 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.199897 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.199952 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:53.202917 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44615: BlockingRecv error: recv got EOF from 127.0.98.3:44615 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 02:42:53 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:53 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:53.962026 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:53.956935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:53567)
0927 02:42:53.957088 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:53.957099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:53.957163 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:53.957339 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:53.957340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:53.957497 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:53.957546 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:53.957548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.958310 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.958312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:53.958906 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:53.958908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.959337 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.959338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:53.959401 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:53.959957 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:53.959962 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:53.961963 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
W20260927 02:42:53.962240 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:53.957052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53567 (local address 127.0.98.62:45297)
0927 02:42:53.957218 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:53.957220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:53.957230 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:53.957253 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:53.957254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:53.957264 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:53.957325 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:53.957649 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.958203 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.959009 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:53.959242 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:53.959309 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:53.962089 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53567: BlockingRecv error: recv got EOF from 127.0.98.1:53567 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":42,"threads_started":1}
Sep 27 02:42:54 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:54 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:54.090866 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:54.085922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:38385)
0927 02:42:54.086081 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:54.086093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:54.086230 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:54.086455 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:54.086456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:54.086623 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:54.086671 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:54.086672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.087321 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.087322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:54.087873 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:54.087875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.088240 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.088241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:54.088283 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:54.088746 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:54.088751 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:54.090784 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
W20260927 02:42:54.091034 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:54.086050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38385 (local address 127.0.98.62:45297)
0927 02:42:54.086113 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:54.086115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:54.086211 (+ 96us) server_negotiation.cc:407] Connection header received
0927 02:42:54.086338 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:54.086339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:54.086351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:54.086411 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:54.086743 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.087246 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.087967 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.088164 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.088228 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:54.090959 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38385: BlockingRecv error: recv got EOF from 127.0.98.2:38385 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 02:42:54 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:54 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:54.207930 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:54.203290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:36361)
0927 02:42:54.203461 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:54.203474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:54.203551 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:54.203723 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:54.203724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:54.203883 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:54.203938 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:54.203944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.204586 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.204587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:54.205215 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:54.205217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.205630 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.205630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:54.205677 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:54.206233 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:54.206237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:54.207863 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
W20260927 02:42:54.208051 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:54.203383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36361 (local address 127.0.98.62:45297)
0927 02:42:54.203465 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:54.203467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:54.203541 (+ 74us) server_negotiation.cc:407] Connection header received
0927 02:42:54.203620 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:54.203621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:54.203633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:54.203681 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:54.204008 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.204514 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.205317 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.205513 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.205600 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:54.207980 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36361: BlockingRecv error: recv got EOF from 127.0.98.3:36361 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 02:42:54 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:54 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:54.967497 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:54.962603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:52205)
0927 02:42:54.962757 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:54.962771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:54.962841 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:54.963059 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:54.963060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:54.963219 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:54.963268 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:54.963270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.963977 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.963978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:54.964573 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:54.964576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.964937 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.964938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:54.964989 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:54.965499 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:54.965504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:54.967423 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
W20260927 02:42:54.967634 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:54.962713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52205 (local address 127.0.98.62:45297)
0927 02:42:54.962864 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:54.962866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:54.962905 (+ 39us) server_negotiation.cc:407] Connection header received
0927 02:42:54.962934 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:54.962935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:54.962945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:54.963009 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:54.963354 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.963872 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.964651 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:54.964853 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:54.964909 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:54.967555 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52205: BlockingRecv error: recv got EOF from 127.0.98.1:52205 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
Sep 27 02:42:55 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:55 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:55.096220 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:55.091325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:47825)
0927 02:42:55.091460 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:55.091473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:55.091542 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:55.091659 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:55.091660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:55.091828 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:55.091874 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:55.091876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.092633 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.092634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:55.093169 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:55.093171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.093593 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.093594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:55.093649 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:55.094119 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:55.094123 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:55.096156 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260927 02:42:55.096403 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:55.091384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47825 (local address 127.0.98.62:45297)
0927 02:42:55.091477 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:55.091479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:55.091533 (+ 54us) server_negotiation.cc:407] Connection header received
0927 02:42:55.091596 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:55.091597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:55.091607 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:55.091654 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:55.091946 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.092561 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.093287 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.093532 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.093587 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:55.096275 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47825: BlockingRecv error: recv got EOF from 127.0.98.2:47825 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 02:42:55 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:55 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:55.213644 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:55.208510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:48019)
0927 02:42:55.208662 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:55.208675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:55.208740 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:55.208898 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:55.208899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:55.209183 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:55.209247 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:55.209250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.210042 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.210043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:55.210658 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:55.210660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.211047 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.211048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:55.211144 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:55.211646 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:55.211651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:55.213565 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
W20260927 02:42:55.213793 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:55.208634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48019 (local address 127.0.98.62:45297)
0927 02:42:55.208740 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:55.208742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:55.208752 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:42:55.208801 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:55.208802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:55.208817 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:55.208872 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:55.209344 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.209948 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.210767 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.210965 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.211028 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:55.213672 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48019: BlockingRecv error: recv got EOF from 127.0.98.3:48019 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 02:42:55 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:55 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:55.972894 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:55.968002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:40769)
0927 02:42:55.968146 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:55.968158 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:55.968229 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:55.968409 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:55.968410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:55.968578 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:55.968649 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:55.968650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.969312 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.969313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:55.969872 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:55.969874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.970228 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.970229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:55.970272 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:55.970911 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:55.970916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:55.972833 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
W20260927 02:42:55.973003 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:55.968123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40769 (local address 127.0.98.62:45297)
0927 02:42:55.968270 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:55.968272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:55.968281 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:55.968304 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:55.968305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:55.968314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:55.968374 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:55.968726 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.969229 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.969960 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.970158 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.970215 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:55.972933 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40769: BlockingRecv error: recv got EOF from 127.0.98.1:40769 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
Sep 27 02:42:56 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:56 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:56.101615 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:56.096762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:44407)
0927 02:42:56.096910 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:56.096921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:56.096991 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:56.097193 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:56.097194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:56.097373 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:56.097420 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:56.097422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.098233 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.098233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:56.098755 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:56.098758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.099135 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.099136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:56.099174 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:56.099658 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:56.099662 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:56.101516 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
W20260927 02:42:56.101684 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:56.096828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44407 (local address 127.0.98.62:45297)
0927 02:42:56.096891 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:56.096893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:56.096984 (+ 91us) server_negotiation.cc:407] Connection header received
0927 02:42:56.097102 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:56.097103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:56.097119 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:56.097176 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:56.097478 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.098159 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.098832 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.099070 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.099133 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:56.101635 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44407: BlockingRecv error: recv got EOF from 127.0.98.2:44407 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 02:42:56 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:56 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:56.219085 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:56.214205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:44165)
0927 02:42:56.214355 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:56.214366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:56.214434 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:56.214657 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:56.214659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:56.214827 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:56.214876 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:56.214877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.215598 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.215599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:56.216132 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:56.216133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.216554 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.216555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:56.216615 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:56.217027 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:56.217031 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:56.219028 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
W20260927 02:42:56.219236 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:56.214304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44165 (local address 127.0.98.62:45297)
0927 02:42:56.214391 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:56.214392 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:56.214419 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:42:56.214530 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:56.214531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:56.214545 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:56.214608 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:56.214980 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.215504 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.216225 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.216450 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.216518 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:56.219122 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44165: BlockingRecv error: recv got EOF from 127.0.98.3:44165 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 02:42:56 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:56 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:56.978271 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:56.973400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:40473)
0927 02:42:56.973544 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:56.973554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:56.973603 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:56.973806 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:56.973806 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:56.973962 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:56.974009 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:56.974010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.974714 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.974715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:56.975358 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:56.975362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.975728 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.975729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:56.975773 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:56.976279 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:56.976284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:56.978202 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
W20260927 02:42:56.978374 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:56.973505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40473 (local address 127.0.98.62:45297)
0927 02:42:56.973665 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:56.973667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:56.973676 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:56.973701 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:56.973702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:56.973711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:56.973787 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:56.974104 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.974609 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.975457 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.975655 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.975732 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:56.978301 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40473: BlockingRecv error: recv got EOF from 127.0.98.1:40473 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
Sep 27 02:42:57 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:57 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:57.111071 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:57.102174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:54615)
0927 02:42:57.102340 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:57.102352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:57.102427 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:57.102550 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:57.102550 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:57.102726 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:57.102793 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:57.102795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.103663 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.103664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:57.104243 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:57.104246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.104593 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.104594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:57.104631 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:57.109078 (+ 4447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:57.109084 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:57.111009 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
W20260927 02:42:57.111254 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:57.102284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54615 (local address 127.0.98.62:45297)
0927 02:42:57.102353 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:57.102356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:57.102418 (+ 62us) server_negotiation.cc:407] Connection header received
0927 02:42:57.102473 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:57.102474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:57.102484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:57.102536 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:57.102856 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.103593 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.104319 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.104525 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.104592 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:57.111185 (+ 6593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54615: BlockingRecv error: recv got EOF from 127.0.98.2:54615 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 02:42:57 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:57 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:57.224540 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:57.219636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:37839)
0927 02:42:57.219793 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:57.219804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:57.219879 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:57.220092 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:57.220093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:57.220256 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:57.220306 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:57.220308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.221088 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.221090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:57.221734 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:57.221737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.222086 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.222087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:57.222136 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:57.222545 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:57.222550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:57.224459 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
W20260927 02:42:57.224668 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:57.219723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37839 (local address 127.0.98.62:45297)
0927 02:42:57.219801 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:57.219803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:57.219870 (+ 67us) server_negotiation.cc:407] Connection header received
0927 02:42:57.219969 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:57.219971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:57.219982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:57.220039 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:57.220448 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.220979 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.221805 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.221991 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.222051 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:57.224558 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37839: BlockingRecv error: recv got EOF from 127.0.98.3:37839 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 02:42:57 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:57 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:57.983814 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:57.978760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:45743)
0927 02:42:57.978936 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:57.978949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:57.979004 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:57.979146 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:57.979153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:57.979350 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:57.979398 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:57.979400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.980173 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.980175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:57.980759 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:57.980761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.981234 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.981235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:57.981298 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:57.981845 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:57.981849 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:57.983753 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
W20260927 02:42:57.983930 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:57.978877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45743 (local address 127.0.98.62:45297)
0927 02:42:57.978987 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:57.978989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:57.978997 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:42:57.979060 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:57.979061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:57.979071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:57.979132 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:57.979508 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.980014 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.980857 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:57.981146 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:57.981222 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:57.983851 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45743: BlockingRecv error: recv got EOF from 127.0.98.1:45743 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":48,"threads_started":1}
Sep 27 02:42:58 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:58 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:58.116755 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:58.111610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:57885)
0927 02:42:58.111781 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:58.111795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:58.111877 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:58.112035 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:58.112036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:58.112244 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:58.112290 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:58.112292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.113039 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.113040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:58.113761 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:58.113764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.114198 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.114198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:58.114249 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:58.114704 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:58.114709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:58.116695 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":111,"threads_started":1}
W20260927 02:42:58.116856 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:58.111770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57885 (local address 127.0.98.62:45297)
0927 02:42:58.111838 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:58.111840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:58.111855 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:42:58.111948 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:58.111949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:58.111962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:58.112080 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:58.112372 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.112929 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.113851 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.114107 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.114172 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:58.116798 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57885: BlockingRecv error: recv got EOF from 127.0.98.2:57885 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 02:42:58 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:58 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:58.230783 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:58.225006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:55735)
0927 02:42:58.225202 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:58.225213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:58.225259 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:58.225508 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:58.225509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:58.225680 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:58.225725 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:58.225727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.226493 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.226495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:58.227131 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:58.227135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.227722 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.227723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:58.227771 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:58.228334 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:58.228340 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:58.230703 (+ 2363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
W20260927 02:42:58.230890 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:58.225168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55735 (local address 127.0.98.62:45297)
0927 02:42:58.225244 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:58.225245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:58.225257 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:42:58.225364 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:58.225365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:58.225380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:58.225438 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:58.225860 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.226398 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.227229 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.227604 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.227687 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:58.230831 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55735: BlockingRecv error: recv got EOF from 127.0.98.3:55735 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 02:42:58 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:58 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:58.988698 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:58.984350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:56775)
0927 02:42:58.984501 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:58.984514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:58.984578 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:58.984746 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:58.984747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:58.984923 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:58.984980 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:58.984982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.985683 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.985684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:58.986217 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:58.986219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.986533 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.986534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:58.986574 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:58.987009 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:58.987013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:58.988639 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260927 02:42:58.988842 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:58.984496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56775 (local address 127.0.98.62:45297)
0927 02:42:58.984606 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:58.984608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:58.984617 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:58.984642 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:58.984643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:58.984652 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:58.984712 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:58.985045 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.985615 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.986291 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:58.986478 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:58.986526 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:58.988732 (+ 2206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56775: BlockingRecv error: recv got EOF from 127.0.98.1:56775 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":46,"threads_started":1}
Sep 27 02:42:59 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:59 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:59.121776 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:59.117271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:40113)
0927 02:42:59.117414 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:59.117425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:59.117490 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:59.117741 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:59.117743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:59.117913 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:59.117957 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:59.117958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.118617 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.118618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:59.119146 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:59.119147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.119456 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.119457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:59.119508 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:59.119951 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:59.119956 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:59.121701 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
W20260927 02:42:59.121840 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:59.117326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40113 (local address 127.0.98.62:45297)
0927 02:42:59.117390 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:59.117391 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:59.117482 (+ 91us) server_negotiation.cc:407] Connection header received
0927 02:42:59.117656 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:59.117657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:59.117672 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:59.117719 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:59.118029 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.118535 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.119215 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.119400 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.119452 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:59.121788 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40113: BlockingRecv error: recv got EOF from 127.0.98.2:40113 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 02:42:59 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:59 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:59.236104 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:59.231400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:43741)
0927 02:42:59.231529 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:59.231541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:59.231596 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:59.231720 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:59.231721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:59.231875 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:59.231923 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:59.231924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.232711 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.232712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:59.233247 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:59.233249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.233588 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.233589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:59.233628 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:59.234090 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:59.234095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:59.236025 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
W20260927 02:42:59.236209 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:59.231487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43741 (local address 127.0.98.62:45297)
0927 02:42:59.231570 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:59.231572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:59.231586 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:42:59.231648 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:59.231649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:59.231660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:59.231708 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:59.232002 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.232585 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.233332 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.233525 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.233584 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:59.236142 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43741: BlockingRecv error: recv got EOF from 127.0.98.3:43741 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 02:42:59 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:42:59 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:42:59.994108 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:59.989250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:38405)
0927 02:42:59.989401 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:59.989415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:59.989495 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:59.989713 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:59.989714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:59.989873 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:59.989925 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:59.989926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.990626 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.990627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:59.991215 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:59.991216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.991568 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.991569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:59.991609 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:59.992138 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:59.992145 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:59.994038 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
W20260927 02:42:59.994226 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:59.989395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38405 (local address 127.0.98.62:45297)
0927 02:42:59.989528 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:59.989529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:59.989538 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:42:59.989563 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:59.989564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:59.989575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:59.989637 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:59.990002 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.990533 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.991306 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.991505 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.991559 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:59.994153 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38405: BlockingRecv error: recv got EOF from 127.0.98.1:38405 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
Sep 27 02:43:00 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:00 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:00.127056 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:00.122298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:52527)
0927 02:43:00.122465 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:00.122478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:00.122563 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:00.122711 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:00.122712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:00.122879 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:00.122943 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:00.122945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.123580 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.123581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:00.124168 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:00.124170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.124587 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.124588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:00.124651 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:00.125109 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:00.125112 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:00.126968 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
W20260927 02:43:00.127127 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:00.122426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52527 (local address 127.0.98.62:45297)
0927 02:43:00.122504 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:00.122506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:00.122541 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:43:00.122634 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:00.122635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:00.122646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:00.122694 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:00.123027 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.123506 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.124255 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.124517 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.124570 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:00.127073 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52527: BlockingRecv error: recv got EOF from 127.0.98.2:52527 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 02:43:00 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:00 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:00.241465 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:00.236602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:60905)
0927 02:43:00.236745 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:00.236758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:00.236855 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:00.237045 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:00.237046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:00.237214 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:00.237258 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:00.237260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.237900 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.237901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:00.238452 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:00.238455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.238850 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.238851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:00.238904 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:00.239416 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:00.239421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:00.241369 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
W20260927 02:43:00.241562 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:00.236688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60905 (local address 127.0.98.62:45297)
0927 02:43:00.236751 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:00.236753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:00.236815 (+ 62us) server_negotiation.cc:407] Connection header received
0927 02:43:00.236955 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:00.236956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:00.236973 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:00.237040 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:00.237316 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.237814 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.238562 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.238754 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.238815 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:00.241496 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60905: BlockingRecv error: recv got EOF from 127.0.98.3:60905 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 02:43:00 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:00 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:00.999271 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:00.994579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:44793)
0927 02:43:00.994722 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:00.994734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:00.994795 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:00.994930 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:00.994930 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:00.995098 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:00.995143 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:00.995145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.995905 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.995906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:00.996490 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:00.996493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.996865 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.996867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:00.996917 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:00.997435 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:00.997440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:00.999201 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
W20260927 02:43:00.999423 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:00.994682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44793 (local address 127.0.98.62:45297)
0927 02:43:00.994801 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:00.994802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:00.994836 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:43:00.994857 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:00.994858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:00.994867 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:00.994915 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:00.995285 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.995796 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.996562 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.996764 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.996830 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:00.999340 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44793: BlockingRecv error: recv got EOF from 127.0.98.1:44793 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
Sep 27 02:43:01 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:01 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:01.132512 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:01.127594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:59631)
0927 02:43:01.127751 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:01.127761 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:01.127809 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:01.127961 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:01.127962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:01.128121 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:01.128168 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:01.128170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:01.128910 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:01.128912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:01.129440 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:01.129441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:01.129807 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:01.129808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:01.129858 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:01.130406 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:01.130413 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:01.132441 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":56,"threads_started":1}
W20260927 02:43:01.132611 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:01.127755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59631 (local address 127.0.98.62:45297)
0927 02:43:01.127808 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:01.127809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:01.127817 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:43:01.127846 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:01.127847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:01.127857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:01.127911 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:01.128275 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:01.128821 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:01.129523 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:01.129726 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:01.129782 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:01.132547 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59631: BlockingRecv error: recv got EOF from 127.0.98.2:59631 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 02:43:01 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:01 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:01.246539 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:01.241985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:34815)
0927 02:43:01.242129 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:01.242140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:01.242222 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:01.242355 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:01.242355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:01.242527 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:01.242570 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:01.242571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:01.243367 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:01.243369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:01.243944 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:01.243947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:01.244283 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:01.244284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:01.244339 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:01.244812 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:01.244817 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:01.246461 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260927 02:43:01.246605 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:01.242034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34815 (local address 127.0.98.62:45297)
0927 02:43:01.242088 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:01.242091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:01.242202 (+ 111us) server_negotiation.cc:407] Connection header received
0927 02:43:01.242282 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:01.242283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:01.242297 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:01.242344 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:01.242649 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:01.243292 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:01.244015 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:01.244225 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:01.244278 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:01.246552 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34815: BlockingRecv error: recv got EOF from 127.0.98.3:34815 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 02:43:02 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:02 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:02.004477 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:01.999791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:36801)
0927 02:43:01.999931 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:01.999942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:01.999996 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:02.000119 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:02.000120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:02.000282 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:02.000328 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:02.000329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.001039 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.001040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:02.001627 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:02.001629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.002046 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.002047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:02.002121 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:02.002568 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:02.002573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:02.004380 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
W20260927 02:43:02.004562 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:01.999903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36801 (local address 127.0.98.62:45297)
0927 02:43:02.000010 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:02.000011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:02.000021 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:43:02.000044 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:02.000045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:02.000054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:02.000107 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:02.000430 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.000946 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.001717 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.001920 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.001987 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:02.004497 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36801: BlockingRecv error: recv got EOF from 127.0.98.1:36801 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":43,"threads_started":1}
Sep 27 02:43:02 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:02 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:02.142580 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:02.133024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:37629)
0927 02:43:02.133161 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:02.133174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:02.133236 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:02.133367 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:02.133368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:02.133528 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:02.133573 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:02.133574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.134386 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.134387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:02.134954 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:02.134957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.135383 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.135384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:02.135451 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:02.140652 (+ 5201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:02.140658 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:02.142493 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
W20260927 02:43:02.142678 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:02.133151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37629 (local address 127.0.98.62:45297)
0927 02:43:02.133254 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:02.133256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:02.133268 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:43:02.133297 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:02.133298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:02.133310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:02.133354 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:02.133642 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.134273 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.135040 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.135248 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.135316 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:02.142623 (+ 7307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37629: BlockingRecv error: recv got EOF from 127.0.98.2:37629 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 02:43:02 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:02 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:02.251717 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:02.247105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:45759)
0927 02:43:02.247237 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:02.247247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:02.247310 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:02.247453 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:02.247454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:02.247648 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:02.247694 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:02.247697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.248369 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.248370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:02.249028 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:02.249031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.249426 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.249427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:02.249473 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:02.249908 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:02.249912 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:02.251651 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
W20260927 02:43:02.251801 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:02.247145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45759 (local address 127.0.98.62:45297)
0927 02:43:02.247248 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:02.247250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:02.247343 (+ 93us) server_negotiation.cc:407] Connection header received
0927 02:43:02.247372 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:02.247373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:02.247388 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:02.247439 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:02.247778 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.248274 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.249113 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.249346 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.249403 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:02.251740 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45759: BlockingRecv error: recv got EOF from 127.0.98.3:45759 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 02:43:03 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:03 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:03.009440 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.005031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:36873)
0927 02:43:03.005201 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:03.005214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:03.005268 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:03.005406 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:03.005407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:03.005565 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:03.005612 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.005613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.006337 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.006338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.006895 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.006898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.007208 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.007209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.007245 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.007678 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:03.007682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:03.009373 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
W20260927 02:43:03.009559 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:03.005130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36873 (local address 127.0.98.62:45297)
0927 02:43:03.005237 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:03.005239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:03.005252 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:43:03.005319 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:03.005320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:03.005331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:03.005391 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:03.005724 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.006249 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.006964 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.007154 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.007197 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.009486 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36873: BlockingRecv error: recv got EOF from 127.0.98.1:36873 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":47,"threads_started":1}
Sep 27 02:43:03 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:43:03 dist-test-slave-44kr krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:43:03.147704 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.143185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:33567)
0927 02:43:03.143339 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:03.143350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:03.143418 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:03.143578 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:03.143579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:03.143763 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:03.143811 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.143813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.144504 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.144505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.145068 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.145070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.145408 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.145409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.145448 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.145867 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:03.145870 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:03.147592 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45297: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
W20260927 02:43:03.147742 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:03.143264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33567 (local address 127.0.98.62:45297)
0927 02:43:03.143306 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:03.143308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:03.143412 (+ 104us) server_negotiation.cc:407] Connection header received
0927 02:43:03.143485 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:03.143485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:03.143498 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:03.143560 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:03.143898 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.144421 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.145139 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.145353 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.145405 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.147692 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33567: BlockingRecv error: recv got EOF from 127.0.98.2:33567 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:43:03.175766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 9125
I20260927 02:43:03.179703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 9277
I20260927 02:43:03.183466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 9409
I20260927 02:43:03.186995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 9055
2026-09-27T02:43:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17344 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T02:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:03Z Disabled control of system clock
I20260927 02:43:03.212481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39517
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:43:03.292155 9606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:03.292379 9606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:03.292433 9606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:03.293668 9606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:03.293725 9606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:03.293737 9606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:03.293749 9606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:03.293759 9606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:03.295152 9606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:03.295372 9606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:03.295621 9606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:03.300941 9611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:03.301005 9606 server_base.cc:1034] running on GCE node
W20260927 02:43:03.301064 9614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:03.300963 9612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:03.301384 9606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:03.301704 9606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:03.302884 9606 hybrid_clock.cc:648] HybridClock initialized: now 1790476983302870 us; error 32 us; skew 500 ppm
I20260927 02:43:03.304478 9606 webserver.cc:460] Webserver started at http://127.0.98.62:46863/ using document root <none> and password file <none>
I20260927 02:43:03.304720 9606 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:03.304775 9606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:03.304893 9606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:03.305992 9606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "fb535a1c44af4b67870de52c00566b68"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "4a025b9068134f6436c5cbbd5a1ce632"
server_key_iv: "517732fa75eaffe8758c9a02d9ce0818"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.306349 9606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb535a1c44af4b67870de52c00566b68"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "4a025b9068134f6436c5cbbd5a1ce632"
server_key_iv: "517732fa75eaffe8758c9a02d9ce0818"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.307711 9606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:43:03.308594 9620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.308782 9606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:43:03.308871 9606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "fb535a1c44af4b67870de52c00566b68"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "4a025b9068134f6436c5cbbd5a1ce632"
server_key_iv: "517732fa75eaffe8758c9a02d9ce0818"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.308939 9606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:03.342010 9606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:03.342329 9606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:03.342458 9606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:03.346295 9606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39517
I20260927 02:43:03.346346 9672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39517 every 8 connection(s)
I20260927 02:43:03.346671 9606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:03.346954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 9606
I20260927 02:43:03.347043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:03.347226 392 external_mini_cluster.cc:1428] Setting key 602871ba4239654e1cefe1977036cc18
I20260927 02:43:03.347375 9673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:03.351130 9673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68: Bootstrap starting.
I20260927 02:43:03.351907 9673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:03.352205 9673 log.cc:824] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:03.352970 9673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68: No bootstrap required, opened a new log
I20260927 02:43:03.353229 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.347739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.0.98.62:39517)
0927 02:43:03.347921 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:03.347928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:03.347954 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:43:03.348380 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:03.348412 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:03.348493 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:03.348611 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:03.348854 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.350028 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.350805 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.351501 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.351630 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.352382 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:03.352405 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:03.352405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:03.352606 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:03.352654 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:03.352670 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:03.352769 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:03.352871 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:43:03.352964 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260927 02:43:03.355326 9673 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb535a1c44af4b67870de52c00566b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } }
I20260927 02:43:03.355494 9673 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:03.355533 9673 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb535a1c44af4b67870de52c00566b68, State: Initialized, Role: FOLLOWER
I20260927 02:43:03.355657 9673 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb535a1c44af4b67870de52c00566b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } }
I20260927 02:43:03.355742 9673 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:03.355782 9673 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:03.355820 9673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:03.356485 9673 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb535a1c44af4b67870de52c00566b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } }
I20260927 02:43:03.356626 9673 leader_election.cc:302] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb535a1c44af4b67870de52c00566b68; no voters:
I20260927 02:43:03.356777 9673 leader_election.cc:288] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:03.356839 9678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:03.356948 9678 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [term 1 LEADER]: Becoming Leader. State: Replica: fb535a1c44af4b67870de52c00566b68, State: Running, Role: LEADER
I20260927 02:43:03.357038 9673 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:03.357023 9678 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb535a1c44af4b67870de52c00566b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } }
I20260927 02:43:03.357363 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb535a1c44af4b67870de52c00566b68. Latest consensus state: current_term: 1 leader_uuid: "fb535a1c44af4b67870de52c00566b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb535a1c44af4b67870de52c00566b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } } }
I20260927 02:43:03.357450 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:03.357612 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb535a1c44af4b67870de52c00566b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb535a1c44af4b67870de52c00566b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } } }
I20260927 02:43:03.357673 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:03.358106 9685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:03.358562 9685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:03.360070 9685 catalog_manager.cc:1347] Generated new cluster ID: 1cd674d754df42daaca4288d9d472c69
I20260927 02:43:03.360132 9685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:03.382032 9685 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:03.382603 9685 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:03.389717 9685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb535a1c44af4b67870de52c00566b68: Generated new TSK 0
I20260927 02:43:03.389945 9685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:03.396889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39517
--builtin_ntp_servers=127.0.98.20:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 02:43:03.514046 9697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:03.514331 9697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:03.514361 9697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:03.516110 9697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:03.516170 9697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:03.516206 9697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:43:03.518191 9697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:03.518463 9697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:03.518808 9697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:03.524247 9702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:03.524277 9703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:03.524430 9697 server_base.cc:1034] running on GCE node
W20260927 02:43:03.524273 9705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:03.524775 9697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:03.525000 9697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:03.526137 9697 hybrid_clock.cc:648] HybridClock initialized: now 1790476983526120 us; error 31 us; skew 500 ppm
I20260927 02:43:03.527468 9697 webserver.cc:460] Webserver started at http://127.0.98.1:45323/ using document root <none> and password file <none>
I20260927 02:43:03.527712 9697 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:03.527767 9697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:03.527896 9697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:03.529036 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "b575d65f240b418b8e0b45645511d67a"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "6b55846a9f5e64f71bdb1f330c255b16"
server_key_iv: "32b53b56d2928dc8e4fefce80ff150de"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.529387 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b575d65f240b418b8e0b45645511d67a"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "6b55846a9f5e64f71bdb1f330c255b16"
server_key_iv: "32b53b56d2928dc8e4fefce80ff150de"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.530712 9697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:43:03.531587 9711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.531831 9697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:43:03.531916 9697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "b575d65f240b418b8e0b45645511d67a"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "6b55846a9f5e64f71bdb1f330c255b16"
server_key_iv: "32b53b56d2928dc8e4fefce80ff150de"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.531978 9697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:03.553648 9697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:03.553958 9697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:03.554090 9697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:03.554327 9697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:03.554663 9697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:03.554702 9697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.554733 9697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:03.554749 9697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.560166 9697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43775
I20260927 02:43:03.560246 9824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43775 every 8 connection(s)
I20260927 02:43:03.560577 9697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:43:03.563274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 9697
I20260927 02:43:03.563359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:43:03.563548 392 external_mini_cluster.cc:1428] Setting key 417fae40b5744edd31f13519260f713c
I20260927 02:43:03.564599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39517
--builtin_ntp_servers=127.0.98.20:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:43:03.566584 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.561262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49615 (local address 127.0.98.62:39517)
0927 02:43:03.561305 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:03.561307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:03.561552 (+ 245us) server_negotiation.cc:407] Connection header received
0927 02:43:03.562070 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:03.562071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:03.562146 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:03.562206 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:03.562893 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.563578 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.564990 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.565279 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.565354 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.566227 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:03.566228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:03.566228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:03.566233 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:03.566247 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:03.566251 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:03.566291 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:03.566468 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 02:43:03.566507 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:43:03.566754 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.561256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.98.1:49615)
0927 02:43:03.561432 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:03.561467 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:03.561975 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:03.562292 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:03.562295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:03.562304 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:03.562803 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.562807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.563677 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.563679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.564904 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.564907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.565392 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.565393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.565517 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.566066 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:43:03.566067 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:03.566107 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:43:03.566138 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:03.566304 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:03.566308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:03.566311 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:43:03.566375 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:43:03.566486 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260927 02:43:03.567224 9825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39517
I20260927 02:43:03.567327 9825 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:03.567512 9825 heartbeater.cc:511] Master 127.0.98.62:39517 requested a full tablet report, sending...
I20260927 02:43:03.568145 9637 ts_manager.cc:194] Registered new tserver with Master: b575d65f240b418b8e0b45645511d67a (127.0.98.1:43775)
I20260927 02:43:03.568813 9637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49615
W20260927 02:43:03.645192 9828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:03.645417 9828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:03.645447 9828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:03.646682 9828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:03.646726 9828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:03.646761 9828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:43:03.648087 9828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:03.648319 9828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:03.648631 9828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:03.653561 9833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:03.653646 9836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:03.653834 9828 server_base.cc:1034] running on GCE node
W20260927 02:43:03.653537 9834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:03.654163 9828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:03.654418 9828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:03.655583 9828 hybrid_clock.cc:648] HybridClock initialized: now 1790476983655555 us; error 45 us; skew 500 ppm
I20260927 02:43:03.656970 9828 webserver.cc:460] Webserver started at http://127.0.98.2:41017/ using document root <none> and password file <none>
I20260927 02:43:03.657251 9828 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:03.657310 9828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:03.657446 9828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:03.658638 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "8bd7596ac3da44d2a24149d6e5177f74"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "9f89ebf41dc21be8c5f44e77d869035d"
server_key_iv: "d88a16b8755a10e5e930a27e673a9467"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.659013 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8bd7596ac3da44d2a24149d6e5177f74"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "9f89ebf41dc21be8c5f44e77d869035d"
server_key_iv: "d88a16b8755a10e5e930a27e673a9467"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.660339 9828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260927 02:43:03.661237 9842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.661399 9828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:43:03.661485 9828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "8bd7596ac3da44d2a24149d6e5177f74"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "9f89ebf41dc21be8c5f44e77d869035d"
server_key_iv: "d88a16b8755a10e5e930a27e673a9467"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.661547 9828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:03.674328 9828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:03.674638 9828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:03.674762 9828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:03.674998 9828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:03.675339 9828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:03.675371 9828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.675400 9828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:03.675424 9828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.681144 9828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35133
I20260927 02:43:03.681192 9955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35133 every 8 connection(s)
I20260927 02:43:03.681517 9828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:43:03.686990 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.682295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59133 (local address 127.0.98.62:39517)
0927 02:43:03.682342 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:03.682344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:03.682584 (+ 240us) server_negotiation.cc:407] Connection header received
0927 02:43:03.683064 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:03.683065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:03.683077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:03.683131 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:03.683861 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.684369 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.685479 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.685678 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.685745 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.686670 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:03.686671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:03.686672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:03.686676 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:03.686692 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:03.686696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:03.686747 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:03.686899 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 02:43:03.686924 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:43:03.687202 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.682289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.98.2:59133)
0927 02:43:03.682457 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:03.682491 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:03.682960 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:03.683214 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:03.683217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:03.683225 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:03.683731 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.683736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.684500 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.684501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.685388 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.685390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.685771 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.685772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.685892 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.686474 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:43:03.686475 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:03.686506 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:43:03.686530 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:03.686778 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:03.686783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:03.686786 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:43:03.686856 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:43:03.686942 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260927 02:43:03.687685 9956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39517
I20260927 02:43:03.687767 9956 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:03.687907 9956 heartbeater.cc:511] Master 127.0.98.62:39517 requested a full tablet report, sending...
I20260927 02:43:03.688241 9637 ts_manager.cc:194] Registered new tserver with Master: 8bd7596ac3da44d2a24149d6e5177f74 (127.0.98.2:35133)
I20260927 02:43:03.688634 9637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59133
I20260927 02:43:03.689908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 9828
I20260927 02:43:03.689989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:43:03.690193 392 external_mini_cluster.cc:1428] Setting key b5a3c1de37e831c2efde645df2432977
I20260927 02:43:03.691074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39517
--builtin_ntp_servers=127.0.98.20:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 02:43:03.771595 9959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:03.771792 9959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:03.771811 9959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:03.772974 9959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:03.773013 9959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:03.773037 9959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:43:03.774317 9959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:03.774514 9959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:03.774788 9959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:03.779663 9964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:03.779716 9967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:03.779732 9965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:03.779830 9959 server_base.cc:1034] running on GCE node
I20260927 02:43:03.780076 9959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:03.780301 9959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:03.781451 9959 hybrid_clock.cc:648] HybridClock initialized: now 1790476983781434 us; error 32 us; skew 500 ppm
I20260927 02:43:03.782716 9959 webserver.cc:460] Webserver started at http://127.0.98.3:38593/ using document root <none> and password file <none>
I20260927 02:43:03.782943 9959 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:03.782989 9959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:03.783123 9959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:03.784147 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "114acf01e82346bc9d325fb3e1512c85"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "80501d734514a62acfd36304c843b4af"
server_key_iv: "ddb9f000834533a046a5429b5842af80"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.784512 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "114acf01e82346bc9d325fb3e1512c85"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "80501d734514a62acfd36304c843b4af"
server_key_iv: "ddb9f000834533a046a5429b5842af80"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.785846 9959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:43:03.786605 9973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.786814 9959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:43:03.786897 9959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "114acf01e82346bc9d325fb3e1512c85"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "80501d734514a62acfd36304c843b4af"
server_key_iv: "ddb9f000834533a046a5429b5842af80"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.786958 9959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:03.801203 9959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:03.801505 9959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:03.801635 9959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:03.801857 9959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:03.802191 9959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:03.802227 9959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.802258 9959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:03.802280 9959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.807871 9959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32931
I20260927 02:43:03.807916 10086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32931 every 8 connection(s)
I20260927 02:43:03.808234 9959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:43:03.813581 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.808982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52337 (local address 127.0.98.62:39517)
0927 02:43:03.809062 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:03.809064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:03.809284 (+ 220us) server_negotiation.cc:407] Connection header received
0927 02:43:03.809788 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:03.809789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:03.809799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:03.809869 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:03.810645 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.811214 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.812225 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.812453 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.812526 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.813348 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:03.813349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:03.813349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:03.813354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:03.813367 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:03.813371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:03.813425 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:03.813485 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 02:43:03.813506 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:43:03.813748 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.808960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.98.3:52337)
0927 02:43:03.809142 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:03.809185 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:03.809668 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:03.809979 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:03.809983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:03.809992 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:03.810544 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.810549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.811297 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.811298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.812130 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:03.812132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.812566 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.812567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:03.812698 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.813203 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:43:03.813203 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:03.813232 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:43:03.813256 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:03.813426 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:03.813429 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:03.813432 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:43:03.813477 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:43:03.813557 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260927 02:43:03.814244 10087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39517
I20260927 02:43:03.814330 10087 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:03.814476 10087 heartbeater.cc:511] Master 127.0.98.62:39517 requested a full tablet report, sending...
I20260927 02:43:03.814823 9637 ts_manager.cc:194] Registered new tserver with Master: 114acf01e82346bc9d325fb3e1512c85 (127.0.98.3:32931)
I20260927 02:43:03.815236 9637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52337
I20260927 02:43:03.815747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 9959
I20260927 02:43:03.815836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:43:03.816049 392 external_mini_cluster.cc:1428] Setting key aa7a37596f3e8c00e5f9492ee2699e85
I20260927 02:43:03.816847 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:43:03.818536 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:03.818062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.0.98.62:39517)
0927 02:43:03.818111 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:03.818112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:03.818250 (+ 138us) server_negotiation.cc:407] Connection header received
0927 02:43:03.818309 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:03.818310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:03.818320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:03.818363 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:03.818479 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60992: BlockingRecv error: recv got EOF from 127.0.0.1:60992 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:43:03.818616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 9697
I20260927 02:43:03.822788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 9828
I20260927 02:43:03.826314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 9959
I20260927 02:43:03.829670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 9606
2026-09-27T02:43:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (642 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T02:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:03Z Disabled control of system clock
I20260927 02:43:03.854139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33663
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:43:03.932193 10098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:03.932427 10098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:03.932448 10098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:03.933558 10098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:03.933595 10098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:03.933606 10098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:03.933616 10098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:03.933626 10098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:03.934932 10098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:03.935137 10098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:03.935377 10098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:03.940049 10103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:03.940049 10106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:03.940179 10104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:03.940404 10098 server_base.cc:1034] running on GCE node
I20260927 02:43:03.940627 10098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:03.940901 10098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:03.942060 10098 hybrid_clock.cc:648] HybridClock initialized: now 1790476983942011 us; error 68 us; skew 500 ppm
I20260927 02:43:03.943428 10098 webserver.cc:460] Webserver started at http://127.0.98.62:45401/ using document root <none> and password file <none>
I20260927 02:43:03.943660 10098 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:03.943713 10098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:03.943848 10098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:03.945008 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "cfaa764e235748f7a2fbbcb86069ef21"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "4e50258f008595ef2985ec102a7eb294"
server_key_iv: "a0f155fe1ef93df670429c485a32cb6c"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.945384 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "cfaa764e235748f7a2fbbcb86069ef21"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "4e50258f008595ef2985ec102a7eb294"
server_key_iv: "a0f155fe1ef93df670429c485a32cb6c"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.946746 10098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:43:03.947614 10112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:03.947836 10098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:43:03.947914 10098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "cfaa764e235748f7a2fbbcb86069ef21"
format_stamp: "Formatted at 2026-09-27 02:43:03 on dist-test-slave-44kr"
server_key: "4e50258f008595ef2985ec102a7eb294"
server_key_iv: "a0f155fe1ef93df670429c485a32cb6c"
server_key_version: "encryptionkey@0"
I20260927 02:43:03.947986 10098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:03.951220 10098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:03.951474 10098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:03.951577 10098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:03.955406 10098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33663
I20260927 02:43:03.955449 10164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33663 every 8 connection(s)
I20260927 02:43:03.955744 10098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:03.956378 10165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:03.958039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 10098
I20260927 02:43:03.958132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:03.958324 392 external_mini_cluster.cc:1428] Setting key 647a0fa52aafbfc503afc63a005498be
I20260927 02:43:03.959437 10165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21: Bootstrap starting.
I20260927 02:43:03.960074 10165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:03.960407 10165 log.cc:824] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:03.960717 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:03.958864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.0.98.62:33663)
0927 02:43:03.959032 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:03.959039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:03.959065 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:43:03.959466 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:03.959473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:03.959512 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:03.959647 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:03.960292 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:03.960295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:03.960296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:03.960299 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:03.960339 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:03.960350 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:03.960432 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:03.960508 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 02:43:03.960568 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260927 02:43:03.961083 10165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21: No bootstrap required, opened a new log
I20260927 02:43:03.962448 10165 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaa764e235748f7a2fbbcb86069ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33663 } }
I20260927 02:43:03.962585 10165 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:03.962605 10165 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfaa764e235748f7a2fbbcb86069ef21, State: Initialized, Role: FOLLOWER
I20260927 02:43:03.962692 10165 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaa764e235748f7a2fbbcb86069ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33663 } }
I20260927 02:43:03.962783 10165 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:03.962822 10165 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:03.962868 10165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:03.963654 10165 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaa764e235748f7a2fbbcb86069ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33663 } }
I20260927 02:43:03.963764 10165 leader_election.cc:302] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfaa764e235748f7a2fbbcb86069ef21; no voters:
I20260927 02:43:03.963925 10165 leader_election.cc:288] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:03.963987 10170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:03.964148 10170 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [term 1 LEADER]: Becoming Leader. State: Replica: cfaa764e235748f7a2fbbcb86069ef21, State: Running, Role: LEADER
I20260927 02:43:03.964164 10165 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:03.964258 10170 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaa764e235748f7a2fbbcb86069ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33663 } }
I20260927 02:43:03.964764 10172 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfaa764e235748f7a2fbbcb86069ef21. Latest consensus state: current_term: 1 leader_uuid: "cfaa764e235748f7a2fbbcb86069ef21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaa764e235748f7a2fbbcb86069ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33663 } } }
I20260927 02:43:03.964761 10171 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfaa764e235748f7a2fbbcb86069ef21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfaa764e235748f7a2fbbcb86069ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33663 } } }
I20260927 02:43:03.964897 10172 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:03.964915 10171 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:03.965691 10178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:03.966359 10178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:03.967830 10178 catalog_manager.cc:1347] Generated new cluster ID: 1833e73cc3a940e4b1c1cb9db8723798
I20260927 02:43:03.967890 10178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:03.982266 10178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:03.982471 10178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:03.989627 10178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfaa764e235748f7a2fbbcb86069ef21: Generated new TSK 0
I20260927 02:43:03.989847 10178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:03.996333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33663
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:43:04.077128 10189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:04.077355 10189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:04.077387 10189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:04.078585 10189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:04.078634 10189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:04.078670 10189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:43:04.080052 10189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:04.080305 10189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:04.080628 10189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:04.086189 10194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:04.086216 10197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:04.086324 10189 server_base.cc:1034] running on GCE node
W20260927 02:43:04.086218 10195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:04.086732 10189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:04.087013 10189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:04.088222 10189 hybrid_clock.cc:648] HybridClock initialized: now 1790476984088209 us; error 31 us; skew 500 ppm
I20260927 02:43:04.089648 10189 webserver.cc:460] Webserver started at http://127.0.98.1:36621/ using document root <none> and password file <none>
I20260927 02:43:04.089879 10189 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:04.089926 10189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:04.090063 10189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:04.091238 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "92bed085033a4ab4b5c7ec47900cebbd"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-44kr"
server_key: "e9b4b653911e31d3660ccf7396c90054"
server_key_iv: "af863203c5e641f33acd4a9b85106e9f"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.091575 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92bed085033a4ab4b5c7ec47900cebbd"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-44kr"
server_key: "e9b4b653911e31d3660ccf7396c90054"
server_key_iv: "af863203c5e641f33acd4a9b85106e9f"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.092962 10189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:43:04.093834 10203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.094029 10189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:43:04.094180 10189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "92bed085033a4ab4b5c7ec47900cebbd"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-44kr"
server_key: "e9b4b653911e31d3660ccf7396c90054"
server_key_iv: "af863203c5e641f33acd4a9b85106e9f"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.094249 10189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:04.097848 10189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:04.098176 10189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:04.098312 10189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:04.098526 10189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:04.098830 10189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:04.098862 10189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.098891 10189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:04.098913 10189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.104658 10189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43615
I20260927 02:43:04.104705 10316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43615 every 8 connection(s)
I20260927 02:43:04.105036 10189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:43:04.108031 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:04.105752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49711 (local address 127.0.98.62:33663)
0927 02:43:04.105839 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:04.105841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:04.106029 (+ 188us) server_negotiation.cc:407] Connection header received
0927 02:43:04.106585 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:04.106586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:04.106593 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:04.106653 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:04.107787 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:04.107789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:04.107789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:04.107793 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:04.107810 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:04.107814 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:04.107864 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:04.107945 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 02:43:04.107959 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:43:04.108254 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:04.105697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33663 (local address 127.0.98.1:49711)
0927 02:43:04.105905 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:04.105941 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:04.106477 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:04.106726 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:04.106730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:04.106738 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:04.107617 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:43:04.107618 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:04.107648 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:43:04.107677 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:04.107870 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:04.107875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:04.107878 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:43:04.107948 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:43:04.108013 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260927 02:43:04.108927 10317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33663
I20260927 02:43:04.109011 10317 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:04.109169 10317 heartbeater.cc:511] Master 127.0.98.62:33663 requested a full tablet report, sending...
I20260927 02:43:04.109612 10129 ts_manager.cc:194] Registered new tserver with Master: 92bed085033a4ab4b5c7ec47900cebbd (127.0.98.1:43615)
I20260927 02:43:04.111229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 10189
I20260927 02:43:04.111322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:43:04.111500 392 external_mini_cluster.cc:1428] Setting key c39e9c79bb341bf94c26e559bce32a7e
I20260927 02:43:04.112592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33663
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:43:04.192075 10320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:04.192279 10320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:04.192301 10320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:04.193483 10320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:04.193527 10320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:04.193552 10320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:43:04.194914 10320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:04.195140 10320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:04.195420 10320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:04.200126 10326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:04.200345 10320 server_base.cc:1034] running on GCE node
W20260927 02:43:04.200120 10328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:04.200126 10325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:04.200651 10320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:04.200876 10320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:04.202033 10320 hybrid_clock.cc:648] HybridClock initialized: now 1790476984202018 us; error 27 us; skew 500 ppm
I20260927 02:43:04.203372 10320 webserver.cc:460] Webserver started at http://127.0.98.2:42443/ using document root <none> and password file <none>
I20260927 02:43:04.203634 10320 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:04.203691 10320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:04.203827 10320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:04.204957 10320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "a0b5bb7615e54a60a23edd574019447c"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-44kr"
server_key: "19108aa6c55ebd47aa3bb9b212bccc51"
server_key_iv: "dff5530ad620aba000759c0c3497df38"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.205355 10320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a0b5bb7615e54a60a23edd574019447c"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-44kr"
server_key: "19108aa6c55ebd47aa3bb9b212bccc51"
server_key_iv: "dff5530ad620aba000759c0c3497df38"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.206713 10320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:43:04.207589 10334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.207768 10320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:43:04.207854 10320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "a0b5bb7615e54a60a23edd574019447c"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-44kr"
server_key: "19108aa6c55ebd47aa3bb9b212bccc51"
server_key_iv: "dff5530ad620aba000759c0c3497df38"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.207916 10320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:04.211298 10320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:04.211556 10320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:04.211674 10320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:04.211870 10320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:04.212175 10320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:04.212206 10320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.212234 10320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:04.212257 10320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.217856 10320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34303
I20260927 02:43:04.217900 10447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34303 every 8 connection(s)
I20260927 02:43:04.218235 10320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:43:04.221403 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:04.218948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46363 (local address 127.0.98.62:33663)
0927 02:43:04.219011 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:04.219013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:04.219323 (+ 310us) server_negotiation.cc:407] Connection header received
0927 02:43:04.219827 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:04.219828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:04.219836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:04.219887 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:04.221123 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:04.221125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:04.221125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:04.221130 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:04.221147 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:04.221151 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:04.221202 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:04.221313 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 02:43:04.221330 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:43:04.221567 10450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:04.218928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33663 (local address 127.0.98.2:46363)
0927 02:43:04.219197 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:04.219232 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:04.219726 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:04.219962 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:04.219966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:04.219974 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:04.220961 (+ 987us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:43:04.220961 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:04.220992 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:43:04.221023 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:04.221209 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:04.221214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:04.221217 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:43:04.221288 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:43:04.221355 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
I20260927 02:43:04.222023 10448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33663
I20260927 02:43:04.222107 10448 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:04.222297 10448 heartbeater.cc:511] Master 127.0.98.62:33663 requested a full tablet report, sending...
I20260927 02:43:04.222682 10129 ts_manager.cc:194] Registered new tserver with Master: a0b5bb7615e54a60a23edd574019447c (127.0.98.2:34303)
I20260927 02:43:04.227351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 10320
I20260927 02:43:04.227468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:43:04.227674 392 external_mini_cluster.cc:1428] Setting key 333aa08cef74976d801193983896e67b
I20260927 02:43:04.228852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33663
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:43:04.312202 10451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:04.312446 10451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:04.312479 10451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:04.313692 10451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:04.313740 10451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:04.313776 10451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:43:04.315153 10451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:04.315357 10451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:04.315616 10451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:04.320710 10456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:04.320732 10459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:04.320742 10457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:04.320884 10451 server_base.cc:1034] running on GCE node
I20260927 02:43:04.321260 10451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:04.321530 10451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:04.322716 10451 hybrid_clock.cc:648] HybridClock initialized: now 1790476984322700 us; error 29 us; skew 500 ppm
I20260927 02:43:04.324079 10451 webserver.cc:460] Webserver started at http://127.0.98.3:45035/ using document root <none> and password file <none>
I20260927 02:43:04.324316 10451 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:04.324366 10451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:04.324514 10451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:04.325604 10451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "67ff2d98212148d2be6a0e9c3b179da4"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-44kr"
server_key: "bca375b83c4726719046c0f026b651e7"
server_key_iv: "3331642b7fe4308dd1bee404d98c4dff"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.325953 10451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "67ff2d98212148d2be6a0e9c3b179da4"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-44kr"
server_key: "bca375b83c4726719046c0f026b651e7"
server_key_iv: "3331642b7fe4308dd1bee404d98c4dff"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.327239 10451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:43:04.328043 10465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.328207 10451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:43:04.328287 10451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "67ff2d98212148d2be6a0e9c3b179da4"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-44kr"
server_key: "bca375b83c4726719046c0f026b651e7"
server_key_iv: "3331642b7fe4308dd1bee404d98c4dff"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.328348 10451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:04.331627 10451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:04.331892 10451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:04.332010 10451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:04.332197 10451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:04.332521 10451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:04.332556 10451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.332580 10451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:04.332603 10451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.338114 10451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38837
I20260927 02:43:04.338277 10578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38837 every 8 connection(s)
I20260927 02:43:04.338490 10451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:43:04.341391 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:04.339208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54793 (local address 127.0.98.62:33663)
0927 02:43:04.339289 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:04.339291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:04.339502 (+ 211us) server_negotiation.cc:407] Connection header received
0927 02:43:04.339962 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:04.339964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:04.339968 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:04.340007 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:04.341189 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:04.341190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:04.341191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:04.341195 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:04.341210 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:04.341214 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:04.341272 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:04.341325 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 02:43:04.341337 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:43:04.341572 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:04.339166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33663 (local address 127.0.98.3:54793)
0927 02:43:04.339376 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:04.339412 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:04.339871 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:04.340088 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:04.340092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:04.340100 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:04.341031 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:43:04.341032 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:04.341062 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:43:04.341098 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:04.341273 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:04.341276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:04.341278 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:43:04.341327 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:43:04.341375 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260927 02:43:04.342165 10579 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33663
I20260927 02:43:04.342248 10579 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:04.342397 10579 heartbeater.cc:511] Master 127.0.98.62:33663 requested a full tablet report, sending...
I20260927 02:43:04.342787 10129 ts_manager.cc:194] Registered new tserver with Master: 67ff2d98212148d2be6a0e9c3b179da4 (127.0.98.3:38837)
I20260927 02:43:04.344329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 10451
I20260927 02:43:04.344444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:43:04.344647 392 external_mini_cluster.cc:1428] Setting key 96895f92166d0c5bba6ceada0c9c7bcd
I20260927 02:43:04.345582 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:43:04.347270 10168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:04.346833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.0.98.62:33663)
0927 02:43:04.346886 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:04.346888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:04.346983 (+ 95us) server_negotiation.cc:407] Connection header received
0927 02:43:04.347049 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:04.347050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:04.347055 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:04.347100 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:04.347219 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59976: BlockingRecv error: recv got EOF from 127.0.0.1:59976 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:43:04.347471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 10189
I20260927 02:43:04.352254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 10320
I20260927 02:43:04.356019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 10451
I20260927 02:43:04.359803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 10098
2026-09-27T02:43:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (530 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:04 dist-test-slave-44kr krb5kdc[10589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:04 dist-test-slave-44kr krb5kdc[10589](info): set up 2 sockets
Sep 27 02:43:04 dist-test-slave-44kr krb5kdc[10589](info): commencing operation
krb5kdc: starting...
W20260927 02:43:05.320947 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:05 dist-test-slave-44kr krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:05.439692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:05.518237 10602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:05.518442 10602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:05.518462 10602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:05.519599 10602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:05.519642 10602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:05.519654 10602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:05.519665 10602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:05.519675 10602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:05.521206 10602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:05.521436 10602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:05.521648 10602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:05.526312 10607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:05.526319 10608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:05.526346 10602 server_base.cc:1034] running on GCE node
W20260927 02:43:05.526319 10610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:05.526829 10602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:05.527154 10602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:05.528316 10602 hybrid_clock.cc:648] HybridClock initialized: now 1790476985528299 us; error 31 us; skew 500 ppm
Sep 27 02:43:05 dist-test-slave-44kr krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:05.531428 10602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:05.531978 10602 webserver.cc:460] Webserver started at http://127.0.98.62:39413/ using document root <none> and password file <none>
I20260927 02:43:05.532277 10602 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:05.532364 10602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:05.532617 10602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:05.534009 10602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "82ba3856ffc44aed935162d2a2ad54e0"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-44kr"
server_key: "a070f48ee8050fb75a467ba128bb5d9d"
server_key_iv: "8ef16cb821c8167596f2091a07b04e85"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.534420 10602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "82ba3856ffc44aed935162d2a2ad54e0"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-44kr"
server_key: "a070f48ee8050fb75a467ba128bb5d9d"
server_key_iv: "8ef16cb821c8167596f2091a07b04e85"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.536059 10602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:43:05.537175 10617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:05.537369 10602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:43:05.537446 10602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "82ba3856ffc44aed935162d2a2ad54e0"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-44kr"
server_key: "a070f48ee8050fb75a467ba128bb5d9d"
server_key_iv: "8ef16cb821c8167596f2091a07b04e85"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.537523 10602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:05.554525 10602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:05.557665 10602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:05.557874 10602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:05.563410 10602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40675
I20260927 02:43:05.563450 10669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40675 every 8 connection(s)
I20260927 02:43:05.563858 10602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:05.564296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 10602
I20260927 02:43:05.564448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:05.564667 392 external_mini_cluster.cc:1428] Setting key 8a5adea4c22f259d706c518b029177b7
I20260927 02:43:05.566078 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:05.569164 10670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0: Bootstrap starting.
I20260927 02:43:05.569828 10670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:05.570106 10670 log.cc:824] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:05.570801 10670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0: No bootstrap required, opened a new log
Sep 27 02:43:05 dist-test-slave-44kr krb5kdc[10589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:05.572825 10670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ba3856ffc44aed935162d2a2ad54e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260927 02:43:05.573001 10670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:05.573045 10670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ba3856ffc44aed935162d2a2ad54e0, State: Initialized, Role: FOLLOWER
I20260927 02:43:05.573196 10670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ba3856ffc44aed935162d2a2ad54e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260927 02:43:05.573285 10670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:05.573316 10670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:05.573364 10670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:05.573931 10670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ba3856ffc44aed935162d2a2ad54e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260927 02:43:05.574024 10670 leader_election.cc:302] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82ba3856ffc44aed935162d2a2ad54e0; no voters:
I20260927 02:43:05.574177 10670 leader_election.cc:288] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:05.574219 10675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:05.574350 10675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [term 1 LEADER]: Becoming Leader. State: Replica: 82ba3856ffc44aed935162d2a2ad54e0, State: Running, Role: LEADER
I20260927 02:43:05.574416 10670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:05.574436 10675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ba3856ffc44aed935162d2a2ad54e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260927 02:43:05.574800 10677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82ba3856ffc44aed935162d2a2ad54e0. Latest consensus state: current_term: 1 leader_uuid: "82ba3856ffc44aed935162d2a2ad54e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ba3856ffc44aed935162d2a2ad54e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } } }
I20260927 02:43:05.574877 10677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:05.575029 10676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82ba3856ffc44aed935162d2a2ad54e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ba3856ffc44aed935162d2a2ad54e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } } }
I20260927 02:43:05.575098 10676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:05.575605 10682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:05.576138 10682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:05.576956 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:05.565200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.0.98.62:40675)
0927 02:43:05.565766 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:05.565776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:05.565814 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:43:05.566411 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:05.566420 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:05.566489 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:05.566671 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:05.567423 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.568838 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.569767 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.570410 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.570495 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:05.573186 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:05.573190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:05.573190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:05.573200 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:05.575008 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:05.575358 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:05.575361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:05.575362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:05.575461 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:05.575690 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:05.575691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:05.575691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:05.575986 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:05.576559 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:05.576639 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 02:43:05.576767 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":465,"threads_started":1}
I20260927 02:43:05.577802 10682 catalog_manager.cc:1347] Generated new cluster ID: 8c533baca8b24d0ea3a3a75534103fd4
I20260927 02:43:05.577857 10682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:05.586679 10682 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:05.587399 10682 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:05.593505 10682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0: Generated new TSK 0
I20260927 02:43:05.593703 10682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:43:05.624298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40675
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:05.706593 10696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:05.706812 10696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:05.706843 10696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:05.707970 10696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:05.708010 10696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:05.708045 10696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:43:05.709487 10696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:05.709723 10696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:05.709929 10696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:05.714764 10702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:05.714788 10701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:05.714764 10704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:05.715066 10696 server_base.cc:1034] running on GCE node
I20260927 02:43:05.715270 10696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:05.715510 10696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:05.716670 10696 hybrid_clock.cc:648] HybridClock initialized: now 1790476985716652 us; error 34 us; skew 500 ppm
Sep 27 02:43:05 dist-test-slave-44kr krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:05.719131 10696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:43:05.719551 10696 webserver.cc:460] Webserver started at http://127.0.98.1:33885/ using document root <none> and password file <none>
I20260927 02:43:05.719759 10696 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:05.719808 10696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:05.719908 10696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:05.720950 10696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "e3765b486cca4398b88598d7b3971a70"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-44kr"
server_key: "100d40c1c9f15e6f2989af8530b605b4"
server_key_iv: "1401ad927bae837143bada6b8ddab4d1"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.721287 10696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e3765b486cca4398b88598d7b3971a70"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-44kr"
server_key: "100d40c1c9f15e6f2989af8530b605b4"
server_key_iv: "1401ad927bae837143bada6b8ddab4d1"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.722546 10696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:43:05.723369 10711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:05.723529 10696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:43:05.723603 10696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "e3765b486cca4398b88598d7b3971a70"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-44kr"
server_key: "100d40c1c9f15e6f2989af8530b605b4"
server_key_iv: "1401ad927bae837143bada6b8ddab4d1"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.723670 10696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:05.744629 10696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:05.746600 10696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:05.746724 10696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:05.746935 10696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:05.747228 10696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:05.747260 10696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:05.747282 10696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:05.747294 10696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:05.752977 10696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36833
I20260927 02:43:05.753026 10824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36833 every 8 connection(s)
I20260927 02:43:05.753329 10696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:43:05 dist-test-slave-44kr krb5kdc[10589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:05.760367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 10696
I20260927 02:43:05.760490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:43:05.760691 392 external_mini_cluster.cc:1428] Setting key 3a276aebe3db744503a385af1a9c2f9e
I20260927 02:43:05.762837 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:05.754029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48359 (local address 127.0.98.62:40675)
0927 02:43:05.754088 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:05.754090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:05.754348 (+ 258us) server_negotiation.cc:407] Connection header received
0927 02:43:05.754822 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:05.754823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:05.754834 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:05.754899 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:05.755773 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.756300 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.757283 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.757478 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.757531 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:05.759665 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:05.759666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:05.759666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:05.759673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:05.761152 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:05.761644 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:05.761646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:05.761646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:05.761712 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:05.762140 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:05.762141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:05.762142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:05.762277 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:05.762344 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:05.762632 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 02:43:05.762737 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:43:05.763043 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:05.754006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40675 (local address 127.0.98.1:48359)
0927 02:43:05.754152 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:05.754270 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:05.754722 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:05.754953 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:05.754957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:05.755503 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:05.755692 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:05.755697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.756371 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.756372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:05.757211 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:05.757213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.757580 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.757581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:05.757706 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:05.758222 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:05.758230 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:05.759573 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:05.761262 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:05.761268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:05.761270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:05.761576 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:05.761872 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:05.761873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:05.761873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:05.762010 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:05.762340 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:05.762347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:05.762488 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 02:43:05.762602 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 02:43:05.762795 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260927 02:43:05.763690 10825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40675
I20260927 02:43:05.763792 10825 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:05.763978 10825 heartbeater.cc:511] Master 127.0.98.62:40675 requested a full tablet report, sending...
I20260927 02:43:05.764573 10634 ts_manager.cc:194] Registered new tserver with Master: e3765b486cca4398b88598d7b3971a70 (127.0.98.1:36833)
I20260927 02:43:05.765250 10634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48359
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:43:05.787557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40675
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:05.867987 10830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:05.868216 10830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:05.868247 10830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:05.869468 10830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:05.869521 10830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:05.869557 10830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:43:05.871018 10830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:05.871271 10830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:05.871485 10830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:05.876500 10836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:05.876549 10835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:05.876497 10838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:05.876732 10830 server_base.cc:1034] running on GCE node
I20260927 02:43:05.876886 10830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:05.877104 10830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:05.878264 10830 hybrid_clock.cc:648] HybridClock initialized: now 1790476985878249 us; error 31 us; skew 500 ppm
Sep 27 02:43:05 dist-test-slave-44kr krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:05.880667 10830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:43:05.881130 10830 webserver.cc:460] Webserver started at http://127.0.98.2:42807/ using document root <none> and password file <none>
I20260927 02:43:05.881325 10830 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:05.881405 10830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:05.881561 10830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:05.882670 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "b6157d5dfad54dc2b1361515eb69b7a5"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-44kr"
server_key: "3033d41753086c3c93ae8deb25c91f63"
server_key_iv: "dec0bac38c06dbb30c8a95f6e438f56b"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.882993 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b6157d5dfad54dc2b1361515eb69b7a5"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-44kr"
server_key: "3033d41753086c3c93ae8deb25c91f63"
server_key_iv: "dec0bac38c06dbb30c8a95f6e438f56b"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.884253 10830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:43:05.885125 10845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:05.885324 10830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:43:05.885426 10830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "b6157d5dfad54dc2b1361515eb69b7a5"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-44kr"
server_key: "3033d41753086c3c93ae8deb25c91f63"
server_key_iv: "dec0bac38c06dbb30c8a95f6e438f56b"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.885494 10830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:05.899308 10830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:05.901305 10830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:05.901477 10830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:05.901726 10830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:05.902043 10830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:05.902081 10830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:05.902113 10830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:05.902138 10830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:05.908149 10830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37157
I20260927 02:43:05.908200 10958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37157 every 8 connection(s)
I20260927 02:43:05.908556 10830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:43:05.912551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 10830
I20260927 02:43:05.912663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:43:05.912859 392 external_mini_cluster.cc:1428] Setting key 1a19fe3d79224616b984a7c10fe33549
Sep 27 02:43:05 dist-test-slave-44kr krb5kdc[10589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:05.918749 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:05.909337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39837 (local address 127.0.98.62:40675)
0927 02:43:05.909419 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:05.909422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:05.909607 (+ 185us) server_negotiation.cc:407] Connection header received
0927 02:43:05.910168 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:05.910169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:05.910180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:05.910238 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:05.911207 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.911797 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.913110 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.913382 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.913453 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:05.915994 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:05.915995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:05.915996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:05.916003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:05.917466 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:05.917857 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:05.917858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:05.917858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:05.917920 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:05.918143 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:05.918144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:05.918144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:05.918277 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:05.918346 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:05.918567 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 02:43:05.918643 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:43:05.918831 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:05.909316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40675 (local address 127.0.98.2:39837)
0927 02:43:05.909471 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:05.909512 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:05.909995 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:05.910302 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:05.910306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:05.910858 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:05.911057 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:05.911062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.911880 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.911881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:05.913032 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:05.913035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:05.913481 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:05.913483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:05.913642 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:05.914216 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:05.914227 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:05.915898 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:05.917543 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:05.917549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:05.917551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:05.917775 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:05.917984 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:05.917984 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:05.917985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:05.918086 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:05.918332 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:05.918335 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:05.918446 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 02:43:05.918529 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:43:05.918644 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260927 02:43:05.919273 10959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40675
I20260927 02:43:05.919368 10959 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:05.919536 10959 heartbeater.cc:511] Master 127.0.98.62:40675 requested a full tablet report, sending...
I20260927 02:43:05.919935 10634 ts_manager.cc:194] Registered new tserver with Master: b6157d5dfad54dc2b1361515eb69b7a5 (127.0.98.2:37157)
I20260927 02:43:05.920459 10634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39837
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:43:05.938884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40675
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:06.016654 10964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:06.016865 10964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:06.016893 10964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:06.018030 10964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:06.018069 10964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:06.018102 10964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:43:06.019469 10964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:06.019693 10964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:06.019889 10964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:06.024524 10972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:06.024655 10964 server_base.cc:1034] running on GCE node
W20260927 02:43:06.024525 10969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:06.024533 10970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:06.024904 10964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:06.025120 10964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:06.026257 10964 hybrid_clock.cc:648] HybridClock initialized: now 1790476986026244 us; error 28 us; skew 500 ppm
Sep 27 02:43:06 dist-test-slave-44kr krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:06.028707 10964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:43:06.029127 10964 webserver.cc:460] Webserver started at http://127.0.98.3:38533/ using document root <none> and password file <none>
I20260927 02:43:06.029315 10964 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:06.029359 10964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:06.029479 10964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:06.030452 10964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "0a5f8038c2934d63b4e77dfb06181bda"
format_stamp: "Formatted at 2026-09-27 02:43:06 on dist-test-slave-44kr"
server_key: "d8c54a55cb4b145ec6079bae0fd969b8"
server_key_iv: "ab4dfef91369aee9e76b37f4d80be14e"
server_key_version: "encryptionkey@0"
I20260927 02:43:06.030773 10964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0a5f8038c2934d63b4e77dfb06181bda"
format_stamp: "Formatted at 2026-09-27 02:43:06 on dist-test-slave-44kr"
server_key: "d8c54a55cb4b145ec6079bae0fd969b8"
server_key_iv: "ab4dfef91369aee9e76b37f4d80be14e"
server_key_version: "encryptionkey@0"
I20260927 02:43:06.032003 10964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:43:06.032701 10979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:06.032872 10964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:06.032965 10964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "0a5f8038c2934d63b4e77dfb06181bda"
format_stamp: "Formatted at 2026-09-27 02:43:06 on dist-test-slave-44kr"
server_key: "d8c54a55cb4b145ec6079bae0fd969b8"
server_key_iv: "ab4dfef91369aee9e76b37f4d80be14e"
server_key_version: "encryptionkey@0"
I20260927 02:43:06.033034 10964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:06.042151 10964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:06.044212 10964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:06.044353 10964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:06.044623 10964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:06.044939 10964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:06.044972 10964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:06.044999 10964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:06.045015 10964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:06.050331 10964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45325
I20260927 02:43:06.050400 11092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45325 every 8 connection(s)
I20260927 02:43:06.050675 10964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:43:06.053189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 10964
I20260927 02:43:06.053306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:43:06.053469 392 external_mini_cluster.cc:1428] Setting key f2ef607fe1613e74ec2db18425f34392
Sep 27 02:43:06 dist-test-slave-44kr krb5kdc[10589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:06.059652 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.051412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55595 (local address 127.0.98.62:40675)
0927 02:43:06.051476 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.051478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.051642 (+ 164us) server_negotiation.cc:407] Connection header received
0927 02:43:06.052140 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.052141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.052155 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.052207 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:06.053306 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.053985 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.055086 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.055312 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.055367 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.057275 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:06.057277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:06.057277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:06.057283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:06.058558 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:06.058898 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:06.058899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:06.058899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:06.058949 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:06.059148 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:06.059148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:06.059149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:06.059278 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:06.059326 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.059528 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.059591 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:43:06.059829 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.051349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40675 (local address 127.0.98.3:55595)
0927 02:43:06.051529 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:06.051566 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:06.052049 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:06.052280 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:06.052285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:06.052913 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:06.053167 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.053174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.054101 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.054102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.055011 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.055013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.055373 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.055374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.055466 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.055896 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:06.055903 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:06.057198 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:06.058617 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:06.058620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:06.058621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:06.058823 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:06.059004 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:06.059005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:06.059005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:06.059102 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:06.059324 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:06.059327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:06.059435 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 02:43:06.059511 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:43:06.059625 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260927 02:43:06.060292 11093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40675
I20260927 02:43:06.060370 11093 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:06.060551 11093 heartbeater.cc:511] Master 127.0.98.62:40675 requested a full tablet report, sending...
I20260927 02:43:06.060899 10634 ts_manager.cc:194] Registered new tserver with Master: 0a5f8038c2934d63b4e77dfb06181bda (127.0.98.3:45325)
I20260927 02:43:06.061259 10634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55595
I20260927 02:43:06.065323 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:43:06.073441 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.066808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.0.98.62:40675)
0927 02:43:06.066876 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.066878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.067050 (+ 172us) server_negotiation.cc:407] Connection header received
0927 02:43:06.067126 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.067127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.067137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.067182 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:06.067740 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.068453 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.069173 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.069389 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.069442 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.070876 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:06.070878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:06.070879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:06.070887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:06.072262 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:06.072635 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:06.072636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:06.072637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:06.072712 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:06.072907 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:06.072908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:06.072908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:06.073069 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:06.073169 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.073302 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.073372 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:43:06.075050 10634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:43:06.076840 10634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:43:06.083361 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.079054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37157 (local address 127.0.0.1:52864)
0927 02:43:06.079202 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:06.079280 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:06.079350 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:06.079772 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:06.079773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:06.079775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:06.079805 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.079806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.081171 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.081172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.082207 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.082208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.083082 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.083083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.083170 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.083184 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:43:06.083232 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:43:06.083276 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"spinlock_wait_cycles":384,"thread_start_us":88,"threads_started":1}
I20260927 02:43:06.083496 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.079054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36833 (local address 127.0.0.1:52984)
0927 02:43:06.079337 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:06.079346 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:06.079393 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:06.079687 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:06.079690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:06.079692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:06.079766 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.079768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.080820 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.080821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.082090 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.082092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.083066 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.083067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.083223 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.083229 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:43:06.083299 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:43:06.083343 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260927 02:43:06.083777 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.079373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.0.98.1:36833)
0927 02:43:06.079542 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.079547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.079557 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:43:06.079597 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.079599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.079615 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.079687 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:06.079844 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.080727 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.082179 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.082993 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.083068 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.083146 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.083336 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.083373 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260927 02:43:06.084159 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.079245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45325 (local address 127.0.0.1:37728)
0927 02:43:06.079508 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:06.079518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:06.079571 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:06.079799 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:06.079800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:06.079803 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:06.079855 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.079857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.082272 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.082272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.082949 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.082950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.083935 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.083936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.083997 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.084001 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:43:06.084053 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:43:06.084097 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":43,"threads_started":1}
I20260927 02:43:06.084276 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.079354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.0.98.3:45325)
0927 02:43:06.079515 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.079521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.079563 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:43:06.079638 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.079640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.079655 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.079719 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:06.079929 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.081070 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.083033 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.083849 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.083915 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.083967 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.084155 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.084199 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260927 02:43:06.084050 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.079179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.0.98.2:37157)
0927 02:43:06.079574 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.079578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.079586 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:43:06.079625 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.079627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.079642 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.079775 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:06.079958 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.081066 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.082277 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.082992 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.083596 (+ 604us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.083670 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.083695 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.083767 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":87,"threads_started":1}
I20260927 02:43:06.084846 10759 tablet_service.cc:1467] Processing CreateTablet for tablet b26bc806a2114a72a7ddfdfdac2ed61e (DEFAULT_TABLE table=test-table [id=333c6f0a9dbb4f229c0670926f20fa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:43:06.084939 10893 tablet_service.cc:1467] Processing CreateTablet for tablet b26bc806a2114a72a7ddfdfdac2ed61e (DEFAULT_TABLE table=test-table [id=333c6f0a9dbb4f229c0670926f20fa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:43:06.084939 11027 tablet_service.cc:1467] Processing CreateTablet for tablet b26bc806a2114a72a7ddfdfdac2ed61e (DEFAULT_TABLE table=test-table [id=333c6f0a9dbb4f229c0670926f20fa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:43:06.085183 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26bc806a2114a72a7ddfdfdac2ed61e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:06.085247 10893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26bc806a2114a72a7ddfdfdac2ed61e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:06.085256 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26bc806a2114a72a7ddfdfdac2ed61e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:06.087773 11107 tablet_bootstrap.cc:492] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70: Bootstrap starting.
I20260927 02:43:06.088369 11108 tablet_bootstrap.cc:492] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: Bootstrap starting.
I20260927 02:43:06.088780 11107 tablet_bootstrap.cc:654] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:06.089040 11108 tablet_bootstrap.cc:654] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:06.089329 11107 log.cc:824] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:06.089633 11108 log.cc:824] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:06.090114 11107 tablet_bootstrap.cc:492] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70: No bootstrap required, opened a new log
I20260927 02:43:06.090185 11107 ts_tablet_manager.cc:1397] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:43:06.090209 11108 tablet_bootstrap.cc:492] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: No bootstrap required, opened a new log
I20260927 02:43:06.090256 11108 ts_tablet_manager.cc:1397] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:43:06.092065 11108 raft_consensus.cc:348] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } }
I20260927 02:43:06.092065 11107 raft_consensus.cc:348] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } }
I20260927 02:43:06.092255 11107 raft_consensus.cc:374] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:06.092255 11108 raft_consensus.cc:374] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:06.092290 11107 raft_consensus.cc:729] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3765b486cca4398b88598d7b3971a70, State: Initialized, Role: FOLLOWER
I20260927 02:43:06.092290 11108 raft_consensus.cc:729] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6157d5dfad54dc2b1361515eb69b7a5, State: Initialized, Role: FOLLOWER
I20260927 02:43:06.092444 11107 consensus_queue.cc:260] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } }
I20260927 02:43:06.092444 11108 consensus_queue.cc:260] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } }
I20260927 02:43:06.092736 11108 ts_tablet_manager.cc:1428] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:06.092736 11107 ts_tablet_manager.cc:1428] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 02:43:06.092794 10959 heartbeater.cc:503] Master 127.0.98.62:40675 was elected leader, sending a full tablet report...
I20260927 02:43:06.089926 11109 tablet_bootstrap.cc:492] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: Bootstrap starting.
I20260927 02:43:06.093045 10825 heartbeater.cc:503] Master 127.0.98.62:40675 was elected leader, sending a full tablet report...
I20260927 02:43:06.093456 11109 tablet_bootstrap.cc:654] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:06.094005 11109 log.cc:824] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:06.094695 11109 tablet_bootstrap.cc:492] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: No bootstrap required, opened a new log
I20260927 02:43:06.094758 11109 ts_tablet_manager.cc:1397] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 02:43:06.096131 11109 raft_consensus.cc:348] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } }
I20260927 02:43:06.096278 11109 raft_consensus.cc:374] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:06.096310 11109 raft_consensus.cc:729] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5f8038c2934d63b4e77dfb06181bda, State: Initialized, Role: FOLLOWER
I20260927 02:43:06.096457 11109 consensus_queue.cc:260] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } }
I20260927 02:43:06.096720 11109 ts_tablet_manager.cc:1428] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:43:06.096769 11093 heartbeater.cc:503] Master 127.0.98.62:40675 was elected leader, sending a full tablet report...
W20260927 02:43:06.159197 10960 tablet.cc:2369] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:43:06.163861 11112 raft_consensus.cc:482] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:43:06.163980 11112 raft_consensus.cc:504] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } }
I20260927 02:43:06.164362 11112 leader_election.cc:288] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3765b486cca4398b88598d7b3971a70 (127.0.98.1:36833), 0a5f8038c2934d63b4e77dfb06181bda (127.0.98.3:45325)
I20260927 02:43:06.167474 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.164592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45325 (local address 127.0.98.2:34785)
0927 02:43:06.164645 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:06.164658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:06.164707 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:06.164868 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:06.164869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:06.164870 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:06.164943 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.164945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.165679 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.165680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.166514 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.166517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.167199 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.167200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.167280 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.167285 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:43:06.167349 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:43:06.167391 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:43:06.167501 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.164679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34785 (local address 127.0.98.3:45325)
0927 02:43:06.164731 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.164733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.164742 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:43:06.164768 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.164769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.164780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.164824 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:06.165009 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.165576 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.166613 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.167117 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.167169 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.167179 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.167395 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.167433 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:43:06.167805 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.164646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36833 (local address 127.0.98.2:55845)
0927 02:43:06.164825 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:06.164835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:06.164892 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:06.165023 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:06.165024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:06.165026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:06.165088 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.165090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.165862 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.165863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.166787 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:06.166790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.167494 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.167495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:06.167607 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.167612 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:43:06.167696 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:43:06.167733 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":131,"threads_started":1}
I20260927 02:43:06.167810 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.164744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55845 (local address 127.0.98.1:36833)
0927 02:43:06.164792 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.164794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.164894 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:43:06.164941 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.164942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.164952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.165005 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:06.165180 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.165766 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.166882 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.167406 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.167467 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.167482 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.167711 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.167754 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:43:06.167919 11047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26bc806a2114a72a7ddfdfdac2ed61e" candidate_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a5f8038c2934d63b4e77dfb06181bda" is_pre_election: true
I20260927 02:43:06.168155 11047 raft_consensus.cc:2454] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6157d5dfad54dc2b1361515eb69b7a5 in term 0.
I20260927 02:43:06.168169 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26bc806a2114a72a7ddfdfdac2ed61e" candidate_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3765b486cca4398b88598d7b3971a70" is_pre_election: true
I20260927 02:43:06.168285 10779 raft_consensus.cc:2454] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6157d5dfad54dc2b1361515eb69b7a5 in term 0.
I20260927 02:43:06.168526 10847 leader_election.cc:302] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5f8038c2934d63b4e77dfb06181bda, b6157d5dfad54dc2b1361515eb69b7a5; no voters:
I20260927 02:43:06.168673 11112 raft_consensus.cc:2781] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:43:06.168725 11112 raft_consensus.cc:482] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:43:06.168747 11112 raft_consensus.cc:3037] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:06.169409 11112 raft_consensus.cc:504] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } }
I20260927 02:43:06.169525 11112 leader_election.cc:288] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [CANDIDATE]: Term 1 election: Requested vote from peers e3765b486cca4398b88598d7b3971a70 (127.0.98.1:36833), 0a5f8038c2934d63b4e77dfb06181bda (127.0.98.3:45325)
I20260927 02:43:06.169706 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26bc806a2114a72a7ddfdfdac2ed61e" candidate_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3765b486cca4398b88598d7b3971a70"
I20260927 02:43:06.169719 11047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26bc806a2114a72a7ddfdfdac2ed61e" candidate_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a5f8038c2934d63b4e77dfb06181bda"
I20260927 02:43:06.169787 11047 raft_consensus.cc:3037] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:06.169787 10779 raft_consensus.cc:3037] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:06.170715 11047 raft_consensus.cc:2454] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6157d5dfad54dc2b1361515eb69b7a5 in term 1.
I20260927 02:43:06.170715 10779 raft_consensus.cc:2454] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6157d5dfad54dc2b1361515eb69b7a5 in term 1.
I20260927 02:43:06.170907 10848 leader_election.cc:302] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6157d5dfad54dc2b1361515eb69b7a5, e3765b486cca4398b88598d7b3971a70; no voters:
I20260927 02:43:06.170989 11112 raft_consensus.cc:2781] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:06.171228 11112 raft_consensus.cc:686] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 1 LEADER]: Becoming Leader. State: Replica: b6157d5dfad54dc2b1361515eb69b7a5, State: Running, Role: LEADER
I20260927 02:43:06.171316 11112 consensus_queue.cc:237] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } }
I20260927 02:43:06.172103 10634 catalog_manager.cc:5495] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6157d5dfad54dc2b1361515eb69b7a5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6157d5dfad54dc2b1361515eb69b7a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 } health_report { overall_health: UNKNOWN } } }
I20260927 02:43:06.178934 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.176665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.0.98.2:37157)
0927 02:43:06.176718 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.176720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.176743 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:43:06.176773 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.176774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.176803 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.176860 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:06.176970 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.177477 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.178242 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.178439 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.178491 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.178625 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:06.178737 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:06.178807 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.178827 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.178846 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:43:06.180645 11047 raft_consensus.cc:1261] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [term 1 FOLLOWER]: Refusing update from remote peer b6157d5dfad54dc2b1361515eb69b7a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:43:06.180685 10779 raft_consensus.cc:1261] T b26bc806a2114a72a7ddfdfdac2ed61e P e3765b486cca4398b88598d7b3971a70 [term 1 FOLLOWER]: Refusing update from remote peer b6157d5dfad54dc2b1361515eb69b7a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:43:06.180991 11117 consensus_queue.cc:1035] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3765b486cca4398b88598d7b3971a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:43:06.181186 11112 consensus_queue.cc:1035] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a5f8038c2934d63b4e77dfb06181bda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:43:06.183377 11120 mvcc.cc:205] Tried to move back new op lower bound from 7333793735393214464 to 7333793735358279680. Current Snapshot: MvccSnapshot[applied={T|T < 7333793735393214464}]
I20260927 02:43:06.183444 11122 mvcc.cc:205] Tried to move back new op lower bound from 7333793735393214464 to 7333793735358279680. Current Snapshot: MvccSnapshot[applied={T|T < 7333793735393214464}]
I20260927 02:43:06.184201 11121 mvcc.cc:205] Tried to move back new op lower bound from 7333793735393214464 to 7333793735358279680. Current Snapshot: MvccSnapshot[applied={T|T < 7333793735393214464}]
I20260927 02:43:06.185561 10633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:43:06.185693 10633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:43:06.186357 10633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82ba3856ffc44aed935162d2a2ad54e0: Sending DeleteTablet for 3 replicas of tablet b26bc806a2114a72a7ddfdfdac2ed61e
I20260927 02:43:06.186662 10759 tablet_service.cc:1514] Processing DeleteTablet for tablet b26bc806a2114a72a7ddfdfdac2ed61e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:43:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52984
I20260927 02:43:06.186694 11027 tablet_service.cc:1514] Processing DeleteTablet for tablet b26bc806a2114a72a7ddfdfdac2ed61e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:43:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37728
I20260927 02:43:06.186896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 10696
I20260927 02:43:06.188349 11131 tablet_replica.cc:330] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: stopping tablet replica
I20260927 02:43:06.188478 11131 raft_consensus.cc:2229] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:43:06.188580 11131 raft_consensus.cc:2258] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:43:06.188966 11131 ts_tablet_manager.cc:1905] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:43:06.189302 10893 tablet_service.cc:1514] Processing DeleteTablet for tablet b26bc806a2114a72a7ddfdfdac2ed61e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:43:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52864
I20260927 02:43:06.190562 11132 tablet_replica.cc:330] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: stopping tablet replica
I20260927 02:43:06.190668 11132 raft_consensus.cc:2229] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:43:06.190765 11132 raft_consensus.cc:2258] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:43:06.190800 11131 ts_tablet_manager.cc:1918] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:43:06.190877 11131 log.cc:1196] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/b26bc806a2114a72a7ddfdfdac2ed61e
I20260927 02:43:06.191144 11131 ts_tablet_manager.cc:1939] T b26bc806a2114a72a7ddfdfdac2ed61e P 0a5f8038c2934d63b4e77dfb06181bda: Deleting consensus metadata
I20260927 02:43:06.191183 11132 ts_tablet_manager.cc:1905] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:43:06.191516 10619 catalog_manager.cc:4841] TS 0a5f8038c2934d63b4e77dfb06181bda (127.0.98.3:45325): tablet b26bc806a2114a72a7ddfdfdac2ed61e (table test-table [id=333c6f0a9dbb4f229c0670926f20fa0a]) successfully deleted
I20260927 02:43:06.192692 11132 ts_tablet_manager.cc:1918] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:43:06.192766 11132 log.cc:1196] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/b26bc806a2114a72a7ddfdfdac2ed61e
I20260927 02:43:06.193008 11132 ts_tablet_manager.cc:1939] T b26bc806a2114a72a7ddfdfdac2ed61e P b6157d5dfad54dc2b1361515eb69b7a5: Deleting consensus metadata
I20260927 02:43:06.193488 10619 catalog_manager.cc:4841] TS b6157d5dfad54dc2b1361515eb69b7a5 (127.0.98.2:37157): tablet b26bc806a2114a72a7ddfdfdac2ed61e (table test-table [id=333c6f0a9dbb4f229c0670926f20fa0a]) successfully deleted
W20260927 02:43:06.198889 10620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36833 (error 108)
I20260927 02:43:06.199362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 10830
I20260927 02:43:06.199368 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.199239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36833 (local address 127.0.0.1:52988)
0927 02:43:06.199295 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:06.199333 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 02:43:06.200685 10620 catalog_manager.cc:4584] TS e3765b486cca4398b88598d7b3971a70 (127.0.98.1:36833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b26bc806a2114a72a7ddfdfdac2ed61e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36833: connect: Connection refused (error 111)
I20260927 02:43:06.203557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 10964
I20260927 02:43:06.207394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 10602
2026-09-27T02:43:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1848 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:06 dist-test-slave-44kr krb5kdc[11136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:06 dist-test-slave-44kr krb5kdc[11136](info): set up 2 sockets
Sep 27 02:43:06 dist-test-slave-44kr krb5kdc[11136](info): commencing operation
krb5kdc: starting...
W20260927 02:43:07.165151 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:07 dist-test-slave-44kr krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:43:07.332075 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKApS4J/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:43:09.962 UTC [11166] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:43:09.962 UTC [11166] LOG: listening on IPv4 address "127.0.98.20", port 47577
2026-09-27 02:43:09.967 UTC [11166] LOG: listening on Unix socket "/tmp/.s.PGSQL.47577"
2026-09-27 02:43:09.987 UTC [11169] LOG: database system was shut down at 2026-09-27 02:43:08 UTC
2026-09-27 02:43:09.991 UTC [11166] LOG: database system is ready to accept connections
I20260927 02:43:10.856941 392 mini_postgres.cc:96] Postgres bound to 47577
2026-09-27 02:43:10.862 UTC [11177] FATAL: database "slave" does not exist
127.0.98.20:47577 - accepting connections
I20260927 02:43:10.863278 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:43:10.877117 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:43:11.431967 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:43:11.432034 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-admin
2026-09-27 02:43:11,710 [I] DB FLAVOR :POSTGRES
2026-09-27 02:43:11,711 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:43:11,711 [I] Checking connection..
2026-09-27 02:43:11,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:43:11,956 [I] Checking connection passed.
2026-09-27 02:43:11,956 [I] --------- Verifying version history table ---------
2026-09-27 02:43:11,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:43:12,207 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:43:12,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:43:12,467 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:43:12,468 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:43:12,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:43:12.721 UTC [11241] WARNING: there is no transaction in progress
2026-09-27 02:43:12,726 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:43:12,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:43:12,974 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:43:12,974 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:43:12,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:43:13,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:43:13,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-44kr.c.gcp-upstream.internal','N') ;"
2026-09-27 02:43:13,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:43:13,983 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:43:13,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:43:14,226 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:43:14,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:43:14,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:43:14,736 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:43:14,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:43:15.761 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.780 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.798 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.816 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.834 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.852 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.908 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.915 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.929 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.940 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.950 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.965 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.974 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.981 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.988 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:15.995 UTC [11349] WARNING: there is no transaction in progress
2026-09-27 02:43:16,692 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:43:16,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:43:16,923 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:43:16,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:43:17,183 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:43:17,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:43:17,417 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:43:17,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:43:17,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:43:17,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-44kr.c.gcp-upstream.internal';"
2026-09-27 02:43:18,142 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:43:18,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:43:18,397 [I] DB_PATCHES have already been applied
I20260927 02:43:18.403141 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKApS4J/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:43:18.403209 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:43:18.406183 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39907
Sep 27, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:43:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:43:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:43:19.407832 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.545s user 0.003s sys 0.006s
Sep 27, 2026 2:43:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:43:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 2:43:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:43:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:43:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:43:41.060540 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:43:41 dist-test-slave-44kr krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477021, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:43:41.145623 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:43:41.154122 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:43:42.175411 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:43:42.175480 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-kms
I20260927 02:43:42.175496 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:47577
2026-09-27 02:43:42,343 [I] DB FLAVOR :POSTGRES
2026-09-27 02:43:42,343 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:43:42,343 [I] Checking connection
2026-09-27 02:43:42,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:43:42,586 [I] connection success
2026-09-27 02:43:42,587 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:43:42,587 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:43:42,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:43:42,829 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:43:42,829 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:43:42,829 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:43:42,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47577/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:43:43,117 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:43:43.556741 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:43:43.725781 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:43:43.769382 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:43:43 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477021, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:43:44.191192 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:43:44.191332 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:43:44.191354 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:43:44.194644 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11529
I20260927 02:43:44.194797 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:53643
Sep 27, 2026 2:43:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:43:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:43:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:43:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:43:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:43:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:43:46.211838 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.066s user 0.003s sys 0.147s
I20260927 02:43:46.211944 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:43:46.212024 392 mini_ranger_kms.cc:401] 127.0.98.20:53643/kms/v1/keys
Sep 27 02:43:46 dist-test-slave-44kr krb5kdc[11136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477026, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:43:46 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477026, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:43:47 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477026, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:43:49 dist-test-slave-44kr krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:49.666414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35631
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35631
--ranger_config_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53643/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:49.746773 11580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:49.746984 11580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:49.747004 11580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:49.748152 11580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:49.748189 11580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:49.748202 11580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:49.748214 11580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:49.748224 11580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:49.749712 11580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:53643/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35631
--ranger_config_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:49.749897 11580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:49.750057 11580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:43:49.755091 11580 server_base.cc:1034] running on GCE node
W20260927 02:43:49.755182 11586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:49.755328 11585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:49.755344 11588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:49.755625 11580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:49.755892 11580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:49.757064 11580 hybrid_clock.cc:648] HybridClock initialized: now 1790477029757020 us; error 55 us; skew 500 ppm
Sep 27 02:43:49 dist-test-slave-44kr krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:49.759568 11580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:49.760016 11580 webserver.cc:460] Webserver started at http://127.0.98.62:34665/ using document root <none> and password file <none>
I20260927 02:43:49.760191 11580 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:49.760226 11580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:49.760315 11580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:43:49 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:43:49.822948 11580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "51070f454ecc43c4bfc57103ec7cfa8c"
format_stamp: "Formatted at 2026-09-27 02:43:49 on dist-test-slave-44kr"
server_key: "d1481be08ae6aa3a57516f8a60c91bdc"
server_key_iv: "99684e96dd57c83365366cdfeb06318d"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:49.823309 11580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "51070f454ecc43c4bfc57103ec7cfa8c"
format_stamp: "Formatted at 2026-09-27 02:43:49 on dist-test-slave-44kr"
server_key: "d1481be08ae6aa3a57516f8a60c91bdc"
server_key_iv: "99684e96dd57c83365366cdfeb06318d"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:49.824687 11580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:43:49.840415 11595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:49.840663 11580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:43:49.840773 11580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "51070f454ecc43c4bfc57103ec7cfa8c"
format_stamp: "Formatted at 2026-09-27 02:43:49 on dist-test-slave-44kr"
server_key: "d1481be08ae6aa3a57516f8a60c91bdc"
server_key_iv: "99684e96dd57c83365366cdfeb06318d"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:49.840873 11580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:49.864153 11580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:49.866317 11580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:49.866456 11580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:49.870966 11580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35631
I20260927 02:43:49.871001 11647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35631 every 8 connection(s)
I20260927 02:43:49.871352 11580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:49.871994 11648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:49.872289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 11580
I20260927 02:43:49.872417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:49.872603 392 external_mini_cluster.cc:1428] Setting key fb6231caa0cc80107d7b45a04ae331f6
I20260927 02:43:49.874954 11648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c: Bootstrap starting.
I20260927 02:43:49.875506 11648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:49.875766 11648 log.cc:824] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:49.876528 11648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c: No bootstrap required, opened a new log
I20260927 02:43:49.878849 11648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51070f454ecc43c4bfc57103ec7cfa8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35631 } }
I20260927 02:43:49.879057 11648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:49.879109 11648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51070f454ecc43c4bfc57103ec7cfa8c, State: Initialized, Role: FOLLOWER
I20260927 02:43:49.879261 11648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51070f454ecc43c4bfc57103ec7cfa8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35631 } }
I20260927 02:43:49.879338 11648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:49.879364 11648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:49.879426 11648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:49.880126 11648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51070f454ecc43c4bfc57103ec7cfa8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35631 } }
I20260927 02:43:49.880241 11648 leader_election.cc:302] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51070f454ecc43c4bfc57103ec7cfa8c; no voters:
I20260927 02:43:49.880460 11648 leader_election.cc:288] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:49.880502 11653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 02:43:49 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:49.880751 11653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [term 1 LEADER]: Becoming Leader. State: Replica: 51070f454ecc43c4bfc57103ec7cfa8c, State: Running, Role: LEADER
I20260927 02:43:49.880831 11648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:49.881268 11653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51070f454ecc43c4bfc57103ec7cfa8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35631 } }
I20260927 02:43:49.882205 11648 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:43:49.883407 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51070f454ecc43c4bfc57103ec7cfa8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51070f454ecc43c4bfc57103ec7cfa8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35631 } } }
I20260927 02:43:49.883481 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:49.883576 11655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51070f454ecc43c4bfc57103ec7cfa8c. Latest consensus state: current_term: 1 leader_uuid: "51070f454ecc43c4bfc57103ec7cfa8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51070f454ecc43c4bfc57103ec7cfa8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35631 } } }
I20260927 02:43:49.883666 11669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:49.884168 11669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:49.883673 11655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:49.884732 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:49.873182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.0.98.62:35631)
0927 02:43:49.875160 (+ 1978us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:49.875171 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:49.875209 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:43:49.875797 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:49.875806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:49.875872 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:49.876028 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:49.876713 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:49.877577 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:49.878293 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:49.878698 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:49.878797 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:49.881249 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:49.881253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:49.881253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:49.881265 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:49.882993 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:49.883365 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:49.883368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:49.883368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:49.883473 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:49.883777 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:49.883778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:49.883778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:49.884085 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:49.884230 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:49.884343 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 02:43:49.884492 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1919,"thread_start_us":73,"threads_started":1}
I20260927 02:43:49.886065 11669 catalog_manager.cc:1347] Generated new cluster ID: 23cff9104d9e4020a4a7d0e8a94c843a
I20260927 02:43:49.886112 11669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:49.896446 11669 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:49.897044 11669 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:49.907378 11669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c: Generated new TSK 0
I20260927 02:43:49.907626 11669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:43:50.617130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35631
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:50.704816 11693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:50.705039 11693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:50.705073 11693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:50.706272 11693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:50.706316 11693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:50.706351 11693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:43:50.707810 11693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:50.708043 11693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:50.708252 11693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:50.713069 11698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:50.713184 11693 server_base.cc:1034] running on GCE node
W20260927 02:43:50.713058 11701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:50.713239 11699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:50.713529 11693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:50.713766 11693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:50.714910 11693 hybrid_clock.cc:648] HybridClock initialized: now 1790477030714895 us; error 31 us; skew 500 ppm
Sep 27 02:43:50 dist-test-slave-44kr krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:50.717947 11693 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:43:50.718369 11693 webserver.cc:460] Webserver started at http://127.0.98.1:43115/ using document root <none> and password file <none>
I20260927 02:43:50.718567 11693 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:50.718614 11693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:50.718762 11693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:43:50 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:43:50.732695 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf"
format_stamp: "Formatted at 2026-09-27 02:43:50 on dist-test-slave-44kr"
server_key: "5243aaca8cb8e470e3f3e99a0c447d26"
server_key_iv: "3eefc518e788d8a21427de7db3152bed"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:50.733170 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf"
format_stamp: "Formatted at 2026-09-27 02:43:50 on dist-test-slave-44kr"
server_key: "5243aaca8cb8e470e3f3e99a0c447d26"
server_key_iv: "3eefc518e788d8a21427de7db3152bed"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:50.735338 11693 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:50.752175 11708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:50.752455 11693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260927 02:43:50.752576 11693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf"
format_stamp: "Formatted at 2026-09-27 02:43:50 on dist-test-slave-44kr"
server_key: "5243aaca8cb8e470e3f3e99a0c447d26"
server_key_iv: "3eefc518e788d8a21427de7db3152bed"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:50.752694 11693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:50.767234 11693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:50.769259 11693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:50.769410 11693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:50.769632 11693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:50.769999 11693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:50.770036 11693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:50.770064 11693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:50.770103 11693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:50.776512 11693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38085
I20260927 02:43:50.776880 11693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:43:50.777484 11822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38085 every 8 connection(s)
Sep 27 02:43:50 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:50.785408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 11693
I20260927 02:43:50.785512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:43:50.785959 392 external_mini_cluster.cc:1428] Setting key 786980e0a692ce5ac9d9c3b0266e570c
I20260927 02:43:50.788089 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:50.777873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57723 (local address 127.0.98.62:35631)
0927 02:43:50.778991 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:50.778994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:50.779002 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:43:50.779027 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:50.779028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:50.779040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:50.779114 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:50.779801 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:50.780320 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:50.781368 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:50.781702 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:50.781761 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:50.784628 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:50.784629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:50.784630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:50.784638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:50.786464 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:50.786982 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:50.786984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:50.786984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:50.787047 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:50.787337 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:50.787338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:50.787338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:50.787465 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:50.787521 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:50.787886 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 02:43:50.787980 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":96,"threads_started":1}
I20260927 02:43:50.788297 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:50.777829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35631 (local address 127.0.98.1:57723)
0927 02:43:50.777999 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:50.778065 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:50.778789 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:50.779228 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:50.779233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:50.779481 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:50.779700 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:50.779705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:50.780426 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:50.780427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:50.781285 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:50.781287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:50.781770 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:50.781771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:50.781908 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:50.782442 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:50.782453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:50.784313 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:50.786563 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:50.786572 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:50.786575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:50.786896 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:50.787113 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:50.787114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:50.787114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:50.787261 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:50.787530 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:50.787537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:50.787728 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 02:43:50.787852 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 02:43:50.788019 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260927 02:43:50.789134 11823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35631
I20260927 02:43:50.789243 11823 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:50.789479 11823 heartbeater.cc:511] Master 127.0.98.62:35631 requested a full tablet report, sending...
I20260927 02:43:50.790030 11612 ts_manager.cc:194] Registered new tserver with Master: b2643d8869ba4da3a4c1bc1e5df69bbf (127.0.98.1:38085)
I20260927 02:43:50.790642 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57723
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:43:50.820088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35631
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:50.929814 11830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:50.930168 11830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:50.930239 11830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:50.931520 11830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:50.931618 11830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:50.931671 11830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:43:50.933665 11830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:50.933948 11830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:50.934185 11830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:50.939198 11837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:50.939355 11840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:50.939237 11838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:50.939805 11830 server_base.cc:1034] running on GCE node
I20260927 02:43:50.940019 11830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:50.940260 11830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:50.941440 11830 hybrid_clock.cc:648] HybridClock initialized: now 1790477030941426 us; error 26 us; skew 500 ppm
Sep 27 02:43:50 dist-test-slave-44kr krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:50.944558 11830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:43:50.945082 11830 webserver.cc:460] Webserver started at http://127.0.98.2:41557/ using document root <none> and password file <none>
I20260927 02:43:50.945313 11830 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:50.945366 11830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:50.945509 11830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:43:50 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:43:50.957232 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "b0843d73bf14470fb7ff481e1524c425"
format_stamp: "Formatted at 2026-09-27 02:43:50 on dist-test-slave-44kr"
server_key: "826eab29f716ffb5bec9fe41e5f41d12"
server_key_iv: "68bc5bad21ab4fbdbbb5710d0618dd66"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:50.957640 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b0843d73bf14470fb7ff481e1524c425"
format_stamp: "Formatted at 2026-09-27 02:43:50 on dist-test-slave-44kr"
server_key: "826eab29f716ffb5bec9fe41e5f41d12"
server_key_iv: "68bc5bad21ab4fbdbbb5710d0618dd66"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:50.961308 11830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
Sep 27 02:43:50 dist-test-slave-44kr krb5kdc[11136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:50.985677 11847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:50.985883 11830 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260927 02:43:50.985982 11830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "b0843d73bf14470fb7ff481e1524c425"
format_stamp: "Formatted at 2026-09-27 02:43:50 on dist-test-slave-44kr"
server_key: "826eab29f716ffb5bec9fe41e5f41d12"
server_key_iv: "68bc5bad21ab4fbdbbb5710d0618dd66"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:50.986093 11830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:51.028362 11830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:51.030781 11830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:51.030952 11830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:51.031194 11830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:51.031605 11830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:51.031656 11830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:51.031705 11830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:51.031724 11830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:51.038631 11830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43775
I20260927 02:43:51.040073 11830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:43:51.041999 11960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43775 every 8 connection(s)
I20260927 02:43:51.046962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 11830
I20260927 02:43:51.047087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:43:51.047276 392 external_mini_cluster.cc:1428] Setting key a8448103dd3cd59f94e3d46bcfde3738
Sep 27 02:43:51 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:51.053952 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.042764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35631 (local address 127.0.98.2:51581)
0927 02:43:51.043076 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:51.043131 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:51.043761 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:51.044057 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:51.044062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:51.044311 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:51.044613 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.044620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.045616 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.045617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.046894 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.046897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.047463 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.047464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.047614 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.048167 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:51.048178 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:51.049758 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:51.051924 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:51.051934 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:51.051937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:51.052279 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:51.052529 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:51.052530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:51.052531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:51.052680 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:51.052994 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:51.053001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:51.053193 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 02:43:51.053532 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 02:43:51.053704 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":88,"threads_started":1}
I20260927 02:43:51.054694 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.043068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51581 (local address 127.0.98.62:35631)
0927 02:43:51.043119 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:51.043121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:51.043241 (+ 120us) server_negotiation.cc:407] Connection header received
0927 02:43:51.043892 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:51.043894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:51.043907 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:51.043978 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:51.044710 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.045506 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.047000 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.047337 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.047376 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.049867 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:51.049868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:51.049868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:51.049877 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:51.051788 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:51.052369 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:51.052371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:51.052371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:51.052462 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:51.052762 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:51.052763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:51.052764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:51.052912 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:51.052969 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:51.054497 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0927 02:43:51.054601 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:43:51.055320 11961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35631
I20260927 02:43:51.055429 11961 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:51.055622 11961 heartbeater.cc:511] Master 127.0.98.62:35631 requested a full tablet report, sending...
I20260927 02:43:51.056123 11612 ts_manager.cc:194] Registered new tserver with Master: b0843d73bf14470fb7ff481e1524c425 (127.0.98.2:43775)
I20260927 02:43:51.056663 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51581
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:43:51.087827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35631
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:51.219085 11967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:51.219473 11967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:51.219575 11967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:51.221475 11967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:51.221824 11967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:51.221941 11967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:43:51.224258 11967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:43:51.224798 11967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:51.225158 11967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:51.234695 11972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:51.234704 11975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:51.236119 11967 server_base.cc:1034] running on GCE node
W20260927 02:43:51.236845 11973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:51.237183 11967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:51.237495 11967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:51.238673 11967 hybrid_clock.cc:648] HybridClock initialized: now 1790477031238650 us; error 38 us; skew 500 ppm
Sep 27 02:43:51 dist-test-slave-44kr krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:51.242053 11967 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:43:51.242667 11967 webserver.cc:460] Webserver started at http://127.0.98.3:37909/ using document root <none> and password file <none>
I20260927 02:43:51.242909 11967 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:51.242955 11967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:51.243069 11967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:43:51 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:43:51.264313 11967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "84ec10ede44e4d06bfb9350b339e4e14"
format_stamp: "Formatted at 2026-09-27 02:43:51 on dist-test-slave-44kr"
server_key: "8ff05c4d49fcaecd4ca22807b61ef57c"
server_key_iv: "be0c98fa40da1e3ee7a98f1b73aced61"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:51.264932 11967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "84ec10ede44e4d06bfb9350b339e4e14"
format_stamp: "Formatted at 2026-09-27 02:43:51 on dist-test-slave-44kr"
server_key: "8ff05c4d49fcaecd4ca22807b61ef57c"
server_key_iv: "be0c98fa40da1e3ee7a98f1b73aced61"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:51.267832 11967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 02:43:51.289748 11982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:51.290086 11967 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260927 02:43:51.290195 11967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "84ec10ede44e4d06bfb9350b339e4e14"
format_stamp: "Formatted at 2026-09-27 02:43:51 on dist-test-slave-44kr"
server_key: "8ff05c4d49fcaecd4ca22807b61ef57c"
server_key_iv: "be0c98fa40da1e3ee7a98f1b73aced61"
server_key_version: "kuduclusterkey@0"
I20260927 02:43:51.290306 11967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:51.315301 11967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:51.317608 11967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:51.317763 11967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:51.317992 11967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:43:51.318359 11967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:43:51.318403 11967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:51.318431 11967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:43:51.318452 11967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:51.326691 11967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35087
I20260927 02:43:51.326946 12095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35087 every 8 connection(s)
I20260927 02:43:51.327325 11967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:43:51 dist-test-slave-44kr krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:51.338114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 11967
I20260927 02:43:51.338235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:43:51.338450 392 external_mini_cluster.cc:1428] Setting key a5da766763d684e76688022d9c34df56
I20260927 02:43:51.339463 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.328100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33483 (local address 127.0.98.62:35631)
0927 02:43:51.328174 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:51.328176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:51.328414 (+ 238us) server_negotiation.cc:407] Connection header received
0927 02:43:51.329053 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:51.329055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:51.329067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:51.329094 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:51.330091 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.330823 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.332520 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.332809 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.332858 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.335711 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:51.335713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:51.335713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:51.335723 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:51.337836 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:51.338388 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:51.338390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:51.338390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:51.338453 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:51.338716 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:51.338717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:51.338717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:51.338850 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:51.338891 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:51.339287 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 02:43:51.339370 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:43:51.339962 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.328094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35631 (local address 127.0.98.3:33483)
0927 02:43:51.328256 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:51.328304 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:51.328935 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:51.329290 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:51.329294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:51.329535 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:51.329981 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.329988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.330947 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.330948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.332363 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.332368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.332960 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.332961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.333120 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.333790 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:51.333802 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:51.335571 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:51.337939 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:51.337948 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:51.337951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:51.338289 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:51.338517 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:51.338517 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:51.338518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:51.338653 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:51.338933 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:51.338939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:51.339130 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 02:43:51.339554 (+ 424us) client_negotiation.cc:240] Negotiation successful
0927 02:43:51.339730 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260927 02:43:51.349002 12096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35631
I20260927 02:43:51.349138 12096 heartbeater.cc:464] Registering TS with master...
I20260927 02:43:51.349370 12096 heartbeater.cc:511] Master 127.0.98.62:35631 requested a full tablet report, sending...
I20260927 02:43:51.349754 11612 ts_manager.cc:194] Registered new tserver with Master: 84ec10ede44e4d06bfb9350b339e4e14 (127.0.98.3:35087)
I20260927 02:43:51.350067 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:43:51.350234 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33483
I20260927 02:43:51.359432 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.351593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.0.98.62:35631)
0927 02:43:51.351633 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:51.351635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:51.351895 (+ 260us) server_negotiation.cc:407] Connection header received
0927 02:43:51.351988 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:51.351988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:51.351998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:51.352025 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:51.352839 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.353563 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.354641 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.354959 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.355006 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.356536 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:51.356537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:51.356538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:51.356545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:51.358168 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:51.358590 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:51.358591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:51.358592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:51.358649 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:51.358855 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:51.358855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:51.358856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:51.358986 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:51.359057 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:51.359255 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 02:43:51.359333 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:43:51.361380 11612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:43:51.363737 11612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:43:51.378674 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.367667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35087 (local address 127.0.0.1:43804)
0927 02:43:51.369182 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:51.369190 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:51.369231 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:51.369855 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:51.369858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:51.369860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:51.369943 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.369945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.375700 (+ 5755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.375700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.377303 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.377305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.378352 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.378353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.378455 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.378468 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:43:51.378532 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:43:51.378582 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"thread_start_us":73,"threads_started":1}
I20260927 02:43:51.379451 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.368129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.0.98.3:35087)
0927 02:43:51.368817 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:51.368823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:51.369284 (+ 461us) server_negotiation.cc:407] Connection header received
0927 02:43:51.369332 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:51.369334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:51.369351 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:51.369421 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:51.370026 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.371522 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.377414 (+ 5892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.378192 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.378250 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.378319 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:51.379318 (+ 999us) server_negotiation.cc:299] Negotiation successful
0927 02:43:51.379367 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":75,"threads_started":1}
I20260927 02:43:51.380240 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.368294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.0.98.1:38085)
0927 02:43:51.368881 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:51.368884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:51.369637 (+ 753us) server_negotiation.cc:407] Connection header received
0927 02:43:51.369677 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:51.369679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:51.369695 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:51.369762 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:51.371754 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.373229 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.374769 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.377000 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.377071 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.377140 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:51.380118 (+ 2978us) server_negotiation.cc:299] Negotiation successful
0927 02:43:51.380176 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":48,"threads_started":1}
I20260927 02:43:51.380465 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.367610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.0.98.2:43775)
0927 02:43:51.376506 (+ 8896us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:51.376512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:51.376524 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:43:51.376573 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:51.376575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:51.376592 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:51.376679 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:51.376880 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.377926 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.379129 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.379860 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.379924 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.379996 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:51.380333 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:43:51.380373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8846,"thread_start_us":78,"threads_started":1}
I20260927 02:43:51.380030 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.367817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38085 (local address 127.0.0.1:48910)
0927 02:43:51.369529 (+ 1712us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:51.369538 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:51.369584 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:51.371618 (+ 2034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:51.371619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:51.371622 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:51.371679 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.371681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.373315 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.373316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.374669 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.374673 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.379795 (+ 5122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.379796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.379867 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.379873 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:43:51.379916 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:43:51.379958 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1693,"thread_start_us":52,"threads_started":1}
I20260927 02:43:51.381407 12030 tablet_service.cc:1467] Processing CreateTablet for tablet c55c33dab7d5460fa41579f9bfe50d57 (DEFAULT_TABLE table=test-table [id=f69c06c6037f41588bb1dcbac50808ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:43:51.381815 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55c33dab7d5460fa41579f9bfe50d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:51.381763 11756 tablet_service.cc:1467] Processing CreateTablet for tablet c55c33dab7d5460fa41579f9bfe50d57 (DEFAULT_TABLE table=test-table [id=f69c06c6037f41588bb1dcbac50808ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:43:51.382042 11756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55c33dab7d5460fa41579f9bfe50d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:51.380219 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.367350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43775 (local address 127.0.0.1:51786)
0927 02:43:51.368942 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:51.368992 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:51.369072 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:51.376766 (+ 7694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:51.376768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:51.376769 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:51.376811 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.376812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.378019 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.378020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.379017 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.379020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.380026 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.380027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.380080 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.380084 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:43:51.380130 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:43:51.380162 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1564,"thread_start_us":91,"threads_started":1}
I20260927 02:43:51.387130 11895 tablet_service.cc:1467] Processing CreateTablet for tablet c55c33dab7d5460fa41579f9bfe50d57 (DEFAULT_TABLE table=test-table [id=f69c06c6037f41588bb1dcbac50808ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:43:51.387531 11895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55c33dab7d5460fa41579f9bfe50d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:51.389911 12114 tablet_bootstrap.cc:492] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf: Bootstrap starting.
I20260927 02:43:51.390702 12114 tablet_bootstrap.cc:654] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:51.392621 12114 log.cc:824] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:51.393607 12115 tablet_bootstrap.cc:492] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: Bootstrap starting.
I20260927 02:43:51.394838 12115 tablet_bootstrap.cc:654] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:51.395238 12115 log.cc:824] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:51.395704 12116 tablet_bootstrap.cc:492] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: Bootstrap starting.
I20260927 02:43:51.396745 12116 tablet_bootstrap.cc:654] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:51.397092 12116 log.cc:824] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:51.398630 12114 tablet_bootstrap.cc:492] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf: No bootstrap required, opened a new log
I20260927 02:43:51.398731 12114 ts_tablet_manager.cc:1397] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 02:43:51.401365 12114 raft_consensus.cc:348] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } }
I20260927 02:43:51.401579 12114 raft_consensus.cc:374] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:51.401619 12114 raft_consensus.cc:729] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2643d8869ba4da3a4c1bc1e5df69bbf, State: Initialized, Role: FOLLOWER
I20260927 02:43:51.401798 12114 consensus_queue.cc:260] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } }
I20260927 02:43:51.402099 12114 ts_tablet_manager.cc:1428] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 02:43:51.402256 11823 heartbeater.cc:503] Master 127.0.98.62:35631 was elected leader, sending a full tablet report...
I20260927 02:43:51.403771 12115 tablet_bootstrap.cc:492] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: No bootstrap required, opened a new log
I20260927 02:43:51.403872 12115 ts_tablet_manager.cc:1397] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 02:43:51.406340 12115 raft_consensus.cc:348] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } }
I20260927 02:43:51.406567 12115 raft_consensus.cc:374] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:51.406612 12115 raft_consensus.cc:729] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ec10ede44e4d06bfb9350b339e4e14, State: Initialized, Role: FOLLOWER
I20260927 02:43:51.406781 12115 consensus_queue.cc:260] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } }
I20260927 02:43:51.407150 12115 ts_tablet_manager.cc:1428] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:43:51.407285 12096 heartbeater.cc:503] Master 127.0.98.62:35631 was elected leader, sending a full tablet report...
I20260927 02:43:51.408299 12122 raft_consensus.cc:482] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:43:51.408481 12122 raft_consensus.cc:504] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } }
I20260927 02:43:51.409582 12122 leader_election.cc:288] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84ec10ede44e4d06bfb9350b339e4e14 (127.0.98.3:35087), b0843d73bf14470fb7ff481e1524c425 (127.0.98.2:43775)
I20260927 02:43:51.410894 12116 tablet_bootstrap.cc:492] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: No bootstrap required, opened a new log
I20260927 02:43:51.410972 12116 ts_tablet_manager.cc:1397] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20260927 02:43:51.413031 12116 raft_consensus.cc:348] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } }
I20260927 02:43:51.413218 12116 raft_consensus.cc:374] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:51.413249 12116 raft_consensus.cc:729] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0843d73bf14470fb7ff481e1524c425, State: Initialized, Role: FOLLOWER
I20260927 02:43:51.413365 12116 consensus_queue.cc:260] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } }
I20260927 02:43:51.413631 12116 ts_tablet_manager.cc:1428] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:43:51.413944 11961 heartbeater.cc:503] Master 127.0.98.62:35631 was elected leader, sending a full tablet report...
I20260927 02:43:51.421924 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.409180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33853 (local address 127.0.98.3:35087)
0927 02:43:51.409221 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:51.409223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:51.414841 (+ 5618us) server_negotiation.cc:407] Connection header received
0927 02:43:51.414950 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:51.414951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:51.414964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:51.414999 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:51.415246 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.415999 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.420932 (+ 4933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.421499 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.421550 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.421563 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:51.421803 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 02:43:51.421845 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:43:51.422394 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.409022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35087 (local address 127.0.98.1:33853)
0927 02:43:51.414745 (+ 5723us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:51.414760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:51.414875 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:51.415071 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:51.415072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:51.415074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:51.415165 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.415167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.416501 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.416502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.417783 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.417788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.421624 (+ 3836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.421625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.421712 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.421717 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:43:51.422252 (+ 535us) client_negotiation.cc:240] Negotiation successful
0927 02:43:51.422303 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5697,"thread_start_us":76,"threads_started":1}
I20260927 02:43:51.422883 12050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55c33dab7d5460fa41579f9bfe50d57" candidate_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ec10ede44e4d06bfb9350b339e4e14" is_pre_election: true
I20260927 02:43:51.423062 12050 raft_consensus.cc:2454] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2643d8869ba4da3a4c1bc1e5df69bbf in term 0.
I20260927 02:43:51.423409 11710 leader_election.cc:302] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84ec10ede44e4d06bfb9350b339e4e14, b2643d8869ba4da3a4c1bc1e5df69bbf; no voters:
I20260927 02:43:51.423619 12122 raft_consensus.cc:2781] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:43:51.423743 12122 raft_consensus.cc:482] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:43:51.423820 12122 raft_consensus.cc:3037] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:51.424816 12122 raft_consensus.cc:504] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } }
I20260927 02:43:51.425179 12050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55c33dab7d5460fa41579f9bfe50d57" candidate_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ec10ede44e4d06bfb9350b339e4e14"
I20260927 02:43:51.425258 12050 raft_consensus.cc:3037] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:51.426151 12050 raft_consensus.cc:2454] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2643d8869ba4da3a4c1bc1e5df69bbf in term 1.
I20260927 02:43:51.426395 11710 leader_election.cc:302] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84ec10ede44e4d06bfb9350b339e4e14, b2643d8869ba4da3a4c1bc1e5df69bbf; no voters:
I20260927 02:43:51.426505 12120 raft_consensus.cc:2781] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:51.426568 12122 leader_election.cc:288] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [CANDIDATE]: Term 1 election: Requested vote from peers 84ec10ede44e4d06bfb9350b339e4e14 (127.0.98.3:35087), b0843d73bf14470fb7ff481e1524c425 (127.0.98.2:43775)
I20260927 02:43:51.426688 12120 raft_consensus.cc:686] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [term 1 LEADER]: Becoming Leader. State: Replica: b2643d8869ba4da3a4c1bc1e5df69bbf, State: Running, Role: LEADER
I20260927 02:43:51.426796 12120 consensus_queue.cc:237] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } }
I20260927 02:43:51.427572 11612 catalog_manager.cc:5495] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf reported cstate change: term changed from 0 to 1, leader changed from <none> to b2643d8869ba4da3a4c1bc1e5df69bbf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38085 } health_report { overall_health: HEALTHY } } }
I20260927 02:43:51.428205 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.409524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54017 (local address 127.0.98.2:43775)
0927 02:43:51.409563 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:51.409565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:51.417970 (+ 8405us) server_negotiation.cc:407] Connection header received
0927 02:43:51.418067 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:51.418068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:51.418081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:51.418109 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:43:51.418329 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.419039 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.427065 (+ 8026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.427641 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.427708 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.427721 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:51.428067 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 02:43:51.428118 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:43:51.430251 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.409355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43775 (local address 127.0.98.1:54017)
0927 02:43:51.417914 (+ 8559us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:51.417925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:51.417998 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:51.418173 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:51.418174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:51.418176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:43:51.418248 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.418250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.419161 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.419162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.420814 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:51.420818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.427891 (+ 7073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.427892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:51.427973 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.427977 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:43:51.430098 (+ 2121us) client_negotiation.cc:240] Negotiation successful
0927 02:43:51.430152 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8539,"thread_start_us":83,"threads_started":1}
I20260927 02:43:51.430730 11915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55c33dab7d5460fa41579f9bfe50d57" candidate_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0843d73bf14470fb7ff481e1524c425" is_pre_election: true
I20260927 02:43:51.430882 11915 raft_consensus.cc:2454] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2643d8869ba4da3a4c1bc1e5df69bbf in term 0.
I20260927 02:43:51.430981 11914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55c33dab7d5460fa41579f9bfe50d57" candidate_uuid: "b2643d8869ba4da3a4c1bc1e5df69bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0843d73bf14470fb7ff481e1524c425"
I20260927 02:43:51.431030 11914 raft_consensus.cc:3037] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:51.431835 11914 raft_consensus.cc:2454] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2643d8869ba4da3a4c1bc1e5df69bbf in term 1.
I20260927 02:43:51.434475 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.431194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.0.98.1:38085)
0927 02:43:51.431232 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:51.431233 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:51.431240 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:43:51.431271 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:51.431272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:51.431291 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:51.431354 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:51.431581 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.432280 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.433405 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:51.433670 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:51.433751 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:51.433939 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:51.434073 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:51.434147 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:51.434365 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 02:43:51.434384 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:43:51.436309 11914 raft_consensus.cc:1261] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [term 1 FOLLOWER]: Refusing update from remote peer b2643d8869ba4da3a4c1bc1e5df69bbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:43:51.436311 12050 raft_consensus.cc:1261] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [term 1 FOLLOWER]: Refusing update from remote peer b2643d8869ba4da3a4c1bc1e5df69bbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:43:51.436635 12120 consensus_queue.cc:1035] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0843d73bf14470fb7ff481e1524c425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:43:51.436753 12122 consensus_queue.cc:1035] T c55c33dab7d5460fa41579f9bfe50d57 P b2643d8869ba4da3a4c1bc1e5df69bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "84ec10ede44e4d06bfb9350b339e4e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:43:51.440964 12129 mvcc.cc:205] Tried to move back new op lower bound from 7333793920760709120 to 7333793920724774912. Current Snapshot: MvccSnapshot[applied={T|T < 7333793920760709120}]
I20260927 02:43:51.441624 12130 mvcc.cc:205] Tried to move back new op lower bound from 7333793920760709120 to 7333793920724774912. Current Snapshot: MvccSnapshot[applied={T|T < 7333793920760709120}]
I20260927 02:43:51.444527 11611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:43:51.444682 11611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:43:51.445564 12131 mvcc.cc:205] Tried to move back new op lower bound from 7333793920760709120 to 7333793920724774912. Current Snapshot: MvccSnapshot[applied={T|T < 7333793920760709120}]
I20260927 02:43:51.445828 11611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51070f454ecc43c4bfc57103ec7cfa8c: Sending DeleteTablet for 3 replicas of tablet c55c33dab7d5460fa41579f9bfe50d57
I20260927 02:43:51.446265 11895 tablet_service.cc:1514] Processing DeleteTablet for tablet c55c33dab7d5460fa41579f9bfe50d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:43:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51786
I20260927 02:43:51.446805 12030 tablet_service.cc:1514] Processing DeleteTablet for tablet c55c33dab7d5460fa41579f9bfe50d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:43:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43804
I20260927 02:43:51.447016 11756 tablet_service.cc:1514] Processing DeleteTablet for tablet c55c33dab7d5460fa41579f9bfe50d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:43:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48910
I20260927 02:43:51.447926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 11693
I20260927 02:43:51.449755 12138 tablet_replica.cc:330] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: stopping tablet replica
I20260927 02:43:51.449932 12139 tablet_replica.cc:330] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: stopping tablet replica
I20260927 02:43:51.450169 12138 raft_consensus.cc:2229] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:43:51.450346 12139 raft_consensus.cc:2229] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:43:51.450536 12138 raft_consensus.cc:2258] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:43:51.450691 12139 raft_consensus.cc:2258] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:43:51.451551 12138 ts_tablet_manager.cc:1905] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:43:51.453080 12139 ts_tablet_manager.cc:1905] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:43:51.454777 12139 ts_tablet_manager.cc:1918] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:43:51.454942 12139 log.cc:1196] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/c55c33dab7d5460fa41579f9bfe50d57
I20260927 02:43:51.455562 12139 ts_tablet_manager.cc:1939] T c55c33dab7d5460fa41579f9bfe50d57 P 84ec10ede44e4d06bfb9350b339e4e14: Deleting consensus metadata
I20260927 02:43:51.456077 11597 catalog_manager.cc:4841] TS 84ec10ede44e4d06bfb9350b339e4e14 (127.0.98.3:35087): tablet c55c33dab7d5460fa41579f9bfe50d57 (table test-table [id=f69c06c6037f41588bb1dcbac50808ae]) successfully deleted
I20260927 02:43:51.456516 12138 ts_tablet_manager.cc:1918] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:43:51.456585 12138 log.cc:1196] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/c55c33dab7d5460fa41579f9bfe50d57
I20260927 02:43:51.457090 12138 ts_tablet_manager.cc:1939] T c55c33dab7d5460fa41579f9bfe50d57 P b0843d73bf14470fb7ff481e1524c425: Deleting consensus metadata
I20260927 02:43:51.457496 11596 catalog_manager.cc:4841] TS b0843d73bf14470fb7ff481e1524c425 (127.0.98.2:43775): tablet c55c33dab7d5460fa41579f9bfe50d57 (table test-table [id=f69c06c6037f41588bb1dcbac50808ae]) successfully deleted
W20260927 02:43:51.459154 11597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38085 (error 108)
I20260927 02:43:51.459995 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:51.459876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38085 (local address 127.0.0.1:48924)
0927 02:43:51.459923 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:51.459962 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 02:43:51.460005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 11830
W20260927 02:43:51.460985 11597 catalog_manager.cc:4584] TS b2643d8869ba4da3a4c1bc1e5df69bbf (127.0.98.1:38085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c55c33dab7d5460fa41579f9bfe50d57: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38085: connect: Connection refused (error 111)
I20260927 02:43:51.468308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 11967
I20260927 02:43:51.473742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 11580
I20260927 02:43:51.478399 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:43:51.813503 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:43:51.813603 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:43:52.148622 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:43:52.148 UTC [11166] LOG: received smart shutdown request
2026-09-27 02:43:52.152 UTC [11166] LOG: background worker "logical replication launcher" (PID 11175) exited with exit code 1
2026-09-27 02:43:52.152 UTC [11170] LOG: shutting down
2026-09-27 02:43:52.329 UTC [11166] LOG: database system is shut down
2026-09-27T02:43:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46202 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:52 dist-test-slave-44kr krb5kdc[12150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:52 dist-test-slave-44kr krb5kdc[12150](info): set up 2 sockets
Sep 27 02:43:52 dist-test-slave-44kr krb5kdc[12150](info): commencing operation
krb5kdc: starting...
W20260927 02:43:53.360348 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:53 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:43:53.530812 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKApS4J/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:43:56.266 UTC [12180] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:43:56.266 UTC [12180] LOG: listening on IPv4 address "127.0.98.20", port 33227
2026-09-27 02:43:56.270 UTC [12180] LOG: listening on Unix socket "/tmp/.s.PGSQL.33227"
2026-09-27 02:43:56.290 UTC [12183] LOG: database system was shut down at 2026-09-27 02:43:54 UTC
2026-09-27 02:43:56.295 UTC [12180] LOG: database system is ready to accept connections
I20260927 02:43:57.155967 392 mini_postgres.cc:96] Postgres bound to 33227
2026-09-27 02:43:57.161 UTC [12191] FATAL: database "slave" does not exist
127.0.98.20:33227 - accepting connections
I20260927 02:43:57.162062 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:43:57.176007 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:43:57.726432 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:43:57.726501 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-admin
2026-09-27 02:43:57,997 [I] DB FLAVOR :POSTGRES
2026-09-27 02:43:57,997 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:43:57,997 [I] Checking connection..
2026-09-27 02:43:57,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:43:58,238 [I] Checking connection passed.
2026-09-27 02:43:58,239 [I] --------- Verifying version history table ---------
2026-09-27 02:43:58,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:43:58,482 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:43:58,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:43:58,729 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:43:58,729 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:43:58,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:43:58.971 UTC [12255] WARNING: there is no transaction in progress
2026-09-27 02:43:58,977 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:43:58,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:43:59,217 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:43:59,218 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:43:59,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:43:59,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:43:59,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-44kr.c.gcp-upstream.internal','N') ;"
2026-09-27 02:43:59,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:44:00,196 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:44:00,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:44:00,445 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:44:00,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:44:00,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:44:00,942 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:44:00,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:44:01.986 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.004 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.022 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.039 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.057 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.076 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.141 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.150 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.166 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.177 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.188 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.205 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.218 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.225 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.234 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:02.243 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:44:03,010 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:44:03,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:44:03,260 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:44:03,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:44:03,510 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:44:03,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:44:03,770 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:44:03,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:44:04,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:44:04,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-44kr.c.gcp-upstream.internal';"
2026-09-27 02:44:04,544 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:44:04,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:44:04,795 [I] DB_PATCHES have already been applied
I20260927 02:44:04.801312 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKApS4J/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:44:04.801383 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:44:04.804373 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:47053
Sep 27, 2026 2:44:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:44:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:44:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:44:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:44:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:44:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:44:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:44:05.806432 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.644s user 0.003s sys 0.006s
Sep 27, 2026 2:44:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:44:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:44:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:44:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:44:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:44:27.862249 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:44:27 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:44:27.944248 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:44:27.952751 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:44:28.981957 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:44:28.982026 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-kms
I20260927 02:44:28.982043 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:33227
2026-09-27 02:44:29,136 [I] DB FLAVOR :POSTGRES
2026-09-27 02:44:29,136 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:44:29,136 [I] Checking connection
2026-09-27 02:44:29,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:44:29,369 [I] connection success
2026-09-27 02:44:29,369 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:44:29,369 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:44:29,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:44:29,610 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:44:29,610 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:44:29,610 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:44:29,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:44:29,872 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:44:30.329131 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:44:30.549563 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:44:30.597019 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:44:30 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477067, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:30.950460 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:44:30.950572 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKApS4J/build/release/bin/postgresql.jar:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKApS4J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:44:30.950589 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:44:30.953822 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12543
I20260927 02:44:30.953907 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36711
Sep 27, 2026 2:44:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:44:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:44:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:44:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:44:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:44:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:44:33.015529 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.071s user 0.002s sys 0.134s
I20260927 02:44:33.015640 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:44:33.015690 392 mini_ranger_kms.cc:401] 127.0.98.20:36711/kms/v1/keys
Sep 27 02:44:33 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:44:33 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:44:33 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477073, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:44:36 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:36.352655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40511
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40511
--ranger_config_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36711/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:36.436630 12594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:36.436890 12594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:36.436918 12594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:36.438141 12594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:36.438186 12594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:36.438199 12594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:36.438216 12594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:36.438227 12594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:36.439924 12594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36711/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40511
--ranger_config_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:36.440163 12594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:36.440481 12594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:44:36.445535 12594 server_base.cc:1034] running on GCE node
W20260927 02:44:36.445585 12600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:36.445746 12602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:36.445698 12599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:36.446187 12594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:36.446532 12594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:36.447700 12594 hybrid_clock.cc:648] HybridClock initialized: now 1790477076447680 us; error 38 us; skew 500 ppm
Sep 27 02:44:36 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:36.450767 12594 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:36.451300 12594 webserver.cc:460] Webserver started at http://127.0.98.62:37071/ using document root <none> and password file <none>
I20260927 02:44:36.451538 12594 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:36.451596 12594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:36.451732 12594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:44:36 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:36.497097 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "c26a60fed59f474db4e0d0e1c38e9f0c"
format_stamp: "Formatted at 2026-09-27 02:44:36 on dist-test-slave-44kr"
server_key: "bf73968cebd94e76ba8a45782e47a25a"
server_key_iv: "b9747c8dc6511dde844f4d0a86025052"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:36.497545 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "c26a60fed59f474db4e0d0e1c38e9f0c"
format_stamp: "Formatted at 2026-09-27 02:44:36 on dist-test-slave-44kr"
server_key: "bf73968cebd94e76ba8a45782e47a25a"
server_key_iv: "b9747c8dc6511dde844f4d0a86025052"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:36.499327 12594 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:44:36.512290 12609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:36.512475 12594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:36.512569 12594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "c26a60fed59f474db4e0d0e1c38e9f0c"
format_stamp: "Formatted at 2026-09-27 02:44:36 on dist-test-slave-44kr"
server_key: "bf73968cebd94e76ba8a45782e47a25a"
server_key_iv: "b9747c8dc6511dde844f4d0a86025052"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:36.512670 12594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:36.526950 12594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:36.528940 12594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:36.529083 12594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:36.533449 12594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40511
I20260927 02:44:36.533514 12661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40511 every 8 connection(s)
I20260927 02:44:36.533854 12594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:36.534569 12662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:36.537729 12662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Bootstrap starting.
I20260927 02:44:36.538300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 12594
I20260927 02:44:36.538391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:36.538422 12662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:36.538578 392 external_mini_cluster.cc:1428] Setting key 9559bca6c1f3645c90a06f52046d8870
I20260927 02:44:36.538811 12662 log.cc:824] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:36.539846 12662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: No bootstrap required, opened a new log
I20260927 02:44:36.541533 12662 raft_consensus.cc:348] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } }
I20260927 02:44:36.541682 12662 raft_consensus.cc:374] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:36.541731 12662 raft_consensus.cc:729] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26a60fed59f474db4e0d0e1c38e9f0c, State: Initialized, Role: FOLLOWER
I20260927 02:44:36.541885 12662 consensus_queue.cc:260] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } }
I20260927 02:44:36.541966 12662 raft_consensus.cc:388] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:36.541988 12662 raft_consensus.cc:482] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:36.542033 12662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:36.542547 12662 raft_consensus.cc:504] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } }
I20260927 02:44:36.542642 12662 leader_election.cc:302] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c26a60fed59f474db4e0d0e1c38e9f0c; no voters:
I20260927 02:44:36.542807 12662 leader_election.cc:288] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:36.542842 12667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:36.542981 12667 raft_consensus.cc:686] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 1 LEADER]: Becoming Leader. State: Replica: c26a60fed59f474db4e0d0e1c38e9f0c, State: Running, Role: LEADER
I20260927 02:44:36.543072 12662 sys_catalog.cc:564] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:36.543121 12667 consensus_queue.cc:237] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } }
I20260927 02:44:36.543536 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c26a60fed59f474db4e0d0e1c38e9f0c. Latest consensus state: current_term: 1 leader_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } }
I20260927 02:44:36.543538 12669 sys_catalog.cc:455] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } }
I20260927 02:44:36.543632 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:36.543654 12669 sys_catalog.cc:458] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:36.544548 12674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:36.544770 12662 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:44:36.545034 12674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:44:36 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:36.549642 12674 catalog_manager.cc:1347] Generated new cluster ID: acfd15b591dd4b88af56267d2570d2f9
I20260927 02:44:36.549703 12674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:36.552561 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:36.539022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.0.98.62:40511)
0927 02:44:36.539208 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:36.539218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:36.539306 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:44:36.539856 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:36.539863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:36.539925 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:36.540066 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:36.540720 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:36.542206 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:36.543096 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:36.544587 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:36.544744 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:36.547796 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:36.547851 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:36.547852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:36.547957 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:36.550504 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:36.550912 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:36.550916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:36.550916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:36.551014 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:36.551228 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:36.551229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:36.551229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:36.551558 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:36.551721 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:36.552157 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 02:44:36.552303 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260927 02:44:36.562723 12674 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:36.563503 12674 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:36.567701 12674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Generated new TSK 0
I20260927 02:44:36.567898 12674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:37.244107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40511
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:37.362658 12707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:37.362932 12707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:37.362968 12707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:37.364648 12707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:37.364708 12707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:37.364748 12707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:37.366904 12707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:37.367180 12707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:37.367472 12707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:37.372641 12713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:37.372632 12715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:37.372627 12712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:37.373751 12707 server_base.cc:1034] running on GCE node
I20260927 02:44:37.373957 12707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:37.374189 12707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:37.375324 12707 hybrid_clock.cc:648] HybridClock initialized: now 1790477077375311 us; error 24 us; skew 500 ppm
Sep 27 02:44:37 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:37.377758 12707 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:37.378242 12707 webserver.cc:460] Webserver started at http://127.0.98.1:35341/ using document root <none> and password file <none>
I20260927 02:44:37.378502 12707 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:37.378552 12707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:37.378670 12707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:44:37 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:37.391641 12707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "b09cc3bd10394e17b13e6de3e49f69be"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "bf586d942fa751bcad14a99f264c2385"
server_key_iv: "f4ad8f9f0b59c972cd6badd706e72854"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:37.392180 12707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b09cc3bd10394e17b13e6de3e49f69be"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "bf586d942fa751bcad14a99f264c2385"
server_key_iv: "f4ad8f9f0b59c972cd6badd706e72854"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:37.394109 12707 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:44:37.408440 12722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:37.409060 12707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:37.409173 12707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "b09cc3bd10394e17b13e6de3e49f69be"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "bf586d942fa751bcad14a99f264c2385"
server_key_iv: "f4ad8f9f0b59c972cd6badd706e72854"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:37.409283 12707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:37.443799 12707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:37.449865 12707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:37.450089 12707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:37.450379 12707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:37.450805 12707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:37.450889 12707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:37.450978 12707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:37.451041 12707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:37.457887 12707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41035
I20260927 02:44:37.457984 12837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41035 every 8 connection(s)
I20260927 02:44:37.458443 12707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:37.461575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 12707
I20260927 02:44:37.461695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:37.461869 392 external_mini_cluster.cc:1428] Setting key 957247be058d7b96873e83b50c6609af
Sep 27 02:44:37 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:37.472515 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:37.459179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40511 (local address 127.0.98.1:48305)
0927 02:44:37.459374 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:37.459423 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:37.460050 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:37.460440 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:37.460444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:37.460685 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:37.460946 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:37.460952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:37.462747 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:37.462748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:37.465806 (+ 3058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:37.465811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:37.466355 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:37.466356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:37.466501 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:37.467109 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:37.467120 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:37.468849 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:37.470821 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:37.470829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:37.470832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:37.471135 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:37.471353 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:37.471354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:37.471355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:37.471497 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:37.471806 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:37.471812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:37.471991 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 02:44:37.472090 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:44:37.472253 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260927 02:44:37.473126 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:37.459220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48305 (local address 127.0.98.62:40511)
0927 02:44:37.459450 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:37.459453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:37.459502 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:44:37.460213 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:37.460215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:37.460228 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:37.460289 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:37.461051 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:37.462639 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:37.465906 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:37.466206 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:37.466276 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:37.468960 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:37.468962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:37.468962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:37.468971 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:37.470727 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:37.471220 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:37.471222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:37.471222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:37.471286 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:37.471569 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:37.471570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:37.471570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:37.471717 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:37.471779 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:37.472948 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0927 02:44:37.473033 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260927 02:44:37.473884 12838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40511
I20260927 02:44:37.474022 12838 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:37.474268 12838 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
I20260927 02:44:37.474859 12626 ts_manager.cc:194] Registered new tserver with Master: b09cc3bd10394e17b13e6de3e49f69be (127.0.98.1:41035)
I20260927 02:44:37.475744 12626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48305
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:37.505227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40511
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:44:37 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 02:44:37.638458 12844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:37.638759 12844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:37.638849 12844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:37.640746 12844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:37.640810 12844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:37.640851 12844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:37.643132 12844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:37.643512 12844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:37.643818 12844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:37.649010 12851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:37.651536 12852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:37.652165 12854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:37.652599 12844 server_base.cc:1034] running on GCE node
I20260927 02:44:37.652823 12844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:37.653072 12844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:37.654218 12844 hybrid_clock.cc:648] HybridClock initialized: now 1790477077654197 us; error 40 us; skew 500 ppm
Sep 27 02:44:37 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:37.657394 12844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:37.657923 12844 webserver.cc:460] Webserver started at http://127.0.98.2:45861/ using document root <none> and password file <none>
I20260927 02:44:37.658236 12844 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:37.658367 12844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:37.658556 12844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:44:37 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:37.674651 12844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d52ddba81ef4b15b2a644b8995520e3"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "cf93df5c2ca61c39b0ebc0ea26255729"
server_key_iv: "beb03e8f55450f7134fdb02a8344353a"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:37.675231 12844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d52ddba81ef4b15b2a644b8995520e3"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "cf93df5c2ca61c39b0ebc0ea26255729"
server_key_iv: "beb03e8f55450f7134fdb02a8344353a"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:37.677090 12844 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:37.691383 12861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:37.691661 12844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:37.691781 12844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "4d52ddba81ef4b15b2a644b8995520e3"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "cf93df5c2ca61c39b0ebc0ea26255729"
server_key_iv: "beb03e8f55450f7134fdb02a8344353a"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:37.691879 12844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:37.712211 12844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:37.714637 12844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:37.714876 12844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:37.715193 12844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:37.715622 12844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:37.715723 12844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:37.715795 12844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:37.715852 12844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:37.722453 12844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44567
I20260927 02:44:37.722962 12844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:37.724761 12975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44567 every 8 connection(s)
I20260927 02:44:37.728889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 12844
I20260927 02:44:37.729000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:37.729172 392 external_mini_cluster.cc:1428] Setting key e5b9f576068c36139ac1eac00c0f7d03
Sep 27 02:44:37 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:37.736261 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:37.726056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50507 (local address 127.0.98.62:40511)
0927 02:44:37.726100 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:37.726102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:37.726113 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:44:37.726755 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:37.726757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:37.726769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:37.726798 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:37.727633 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:37.728475 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:37.729910 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:37.730200 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:37.730245 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:37.732896 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:37.732898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:37.732898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:37.732906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:37.734649 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:37.735178 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:37.735180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:37.735180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:37.735237 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:37.735497 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:37.735497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:37.735498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:37.735626 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:37.735668 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:37.736076 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 02:44:37.736166 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:44:37.737035 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:37.725754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40511 (local address 127.0.98.2:50507)
0927 02:44:37.725921 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:37.725966 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:37.726602 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:37.726989 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:37.726993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:37.727241 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:37.727528 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:37.727534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:37.728588 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:37.728589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:37.729817 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:37.729820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:37.730348 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:37.730349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:37.730492 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:37.731090 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:37.731101 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:37.732774 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:37.734756 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:37.734764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:37.734767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:37.735092 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:37.735302 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:37.735303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:37.735303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:37.735431 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:37.735724 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:37.735729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:37.735915 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 02:44:37.736615 (+ 700us) client_negotiation.cc:240] Negotiation successful
0927 02:44:37.736799 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":97,"threads_started":1}
I20260927 02:44:37.737732 12976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40511
I20260927 02:44:37.737829 12976 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:37.738013 12976 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
I20260927 02:44:37.738409 12626 ts_manager.cc:194] Registered new tserver with Master: 4d52ddba81ef4b15b2a644b8995520e3 (127.0.98.2:44567)
I20260927 02:44:37.738914 12626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50507
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:37.772070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40511
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:37.892908 12981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:37.893277 12981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:37.893373 12981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:37.895498 12981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:37.895763 12981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:37.895870 12981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:37.898684 12981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:37.899207 12981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:37.899564 12981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:37.906144 12986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:37.906376 12987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:37.906399 12989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:37.914435 12981 server_base.cc:1034] running on GCE node
I20260927 02:44:37.914774 12981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:37.915123 12981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:37.916291 12981 hybrid_clock.cc:648] HybridClock initialized: now 1790477077916270 us; error 38 us; skew 500 ppm
Sep 27 02:44:37 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:37.919751 12981 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:37.920344 12981 webserver.cc:460] Webserver started at http://127.0.98.3:45261/ using document root <none> and password file <none>
I20260927 02:44:37.920639 12981 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:37.920698 12981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:37.920877 12981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:44:37 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:37.934988 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "da44ab8f61eb4924abd060d53b261b73"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "a3b34ae83e9eebd1747150aadde23dc6"
server_key_iv: "67c3dc403c795d3fa82464c14b39ab88"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:37.935544 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "da44ab8f61eb4924abd060d53b261b73"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "a3b34ae83e9eebd1747150aadde23dc6"
server_key_iv: "67c3dc403c795d3fa82464c14b39ab88"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:37.938210 12981 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:44:37.970645 12996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:37.970937 12981 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.000s sys 0.001s
I20260927 02:44:37.971043 12981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "da44ab8f61eb4924abd060d53b261b73"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "a3b34ae83e9eebd1747150aadde23dc6"
server_key_iv: "67c3dc403c795d3fa82464c14b39ab88"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:37.971155 12981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:38.003722 12981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:38.006338 12981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:38.006600 12981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:38.006966 12981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:38.007392 12981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:38.007483 12981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.007560 12981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:38.007625 12981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.014611 12981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42109
I20260927 02:44:38.015170 12981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:38.021763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 12981
I20260927 02:44:38.021890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:38.022074 392 external_mini_cluster.cc:1428] Setting key 899960c214b4c1fb5e5b7a80f7c817ec
I20260927 02:44:38.016965 13109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42109 every 8 connection(s)
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:38.029024 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.017919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39611 (local address 127.0.98.62:40511)
0927 02:44:38.017967 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:38.017969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:38.018495 (+ 526us) server_negotiation.cc:407] Connection header received
0927 02:44:38.019210 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:38.019211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:38.019223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:38.019253 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:38.020111 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.020941 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.022649 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.022972 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.023014 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.025682 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:38.025683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:38.025684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:38.025691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:38.027363 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.027855 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.027856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.027856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.027916 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.028205 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.028206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.028207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.028343 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:38.028409 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:38.028823 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 02:44:38.028927 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:44:38.029793 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.017918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40511 (local address 127.0.98.3:39611)
0927 02:44:38.018358 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:38.018407 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:38.019083 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:38.019446 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:38.019450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:38.019697 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:38.019998 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.020005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.021061 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.021062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.022437 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.022440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.023114 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.023115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.023258 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.023830 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:38.023840 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:38.025566 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:38.027461 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.027468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.027470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.027756 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.027985 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.027985 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.027986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.028125 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.028457 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:38.028464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:38.028650 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 02:44:38.029375 (+ 725us) client_negotiation.cc:240] Negotiation successful
0927 02:44:38.029540 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":91,"threads_started":1}
I20260927 02:44:38.030520 13110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40511
I20260927 02:44:38.030612 13110 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.030805 13110 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
I20260927 02:44:38.031296 12626 ts_manager.cc:194] Registered new tserver with Master: da44ab8f61eb4924abd060d53b261b73 (127.0.98.3:42109)
I20260927 02:44:38.031769 12626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39611
I20260927 02:44:38.033582 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:44:38.034296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 12707
I20260927 02:44:38.040300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 12844
I20260927 02:44:38.046837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 12981
I20260927 02:44:38.051143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 12594
I20260927 02:44:38.055646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40511
--webserver_interface=127.0.98.62
--webserver_port=37071
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40511
--ranger_config_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36711/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36711/kms with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:38.137244 13113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:38.137467 13113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:38.137490 13113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:38.138695 13113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:38.138759 13113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:38.138773 13113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:38.138784 13113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:38.138795 13113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:38.140484 13113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36711/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40511
--ranger_config_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:38.140713 13113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:38.140935 13113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:38.145972 13118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.146204 13113 server_base.cc:1034] running on GCE node
W20260927 02:44:38.145965 13121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.145965 13119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.146621 13113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:38.146924 13113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:38.148083 13113 hybrid_clock.cc:648] HybridClock initialized: now 1790477078148069 us; error 28 us; skew 500 ppm
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:38.150593 13113 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:38.151039 13113 webserver.cc:460] Webserver started at http://127.0.98.62:37071/ using document root <none> and password file <none>
I20260927 02:44:38.151258 13113 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:38.151309 13113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:38.191987 13113 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:38.192787 13129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.192978 13113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:38.193063 13113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "c26a60fed59f474db4e0d0e1c38e9f0c"
format_stamp: "Formatted at 2026-09-27 02:44:36 on dist-test-slave-44kr"
server_key: "bf73968cebd94e76ba8a45782e47a25a"
server_key_iv: "b9747c8dc6511dde844f4d0a86025052"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:38.193500 13113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:38.201148 13113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:38.203138 13113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:38.203270 13113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:38.207125 13113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40511
I20260927 02:44:38.207209 13181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40511 every 8 connection(s)
I20260927 02:44:38.207525 13113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:38.208870 13182 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:44:38.209482 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Bootstrap starting.
I20260927 02:44:38.211566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 13113
I20260927 02:44:38.211829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41035
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40511
--webserver_port=35341
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36711/kms with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:38.217264 13182 log.cc:824] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:38.218899 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:44:38.219153 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Bootstrap complete.
I20260927 02:44:38.221362 13182 raft_consensus.cc:348] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } }
I20260927 02:44:38.221590 13182 raft_consensus.cc:729] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26a60fed59f474db4e0d0e1c38e9f0c, State: Initialized, Role: FOLLOWER
I20260927 02:44:38.221738 13182 consensus_queue.cc:260] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } }
I20260927 02:44:38.221808 13182 raft_consensus.cc:388] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:38.221855 13182 raft_consensus.cc:482] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:38.221902 13182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:44:38.222919 13182 raft_consensus.cc:504] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } }
I20260927 02:44:38.223016 13182 leader_election.cc:302] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c26a60fed59f474db4e0d0e1c38e9f0c; no voters:
I20260927 02:44:38.223192 13182 leader_election.cc:288] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:44:38.223240 13186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:44:38.223390 13186 raft_consensus.cc:686] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [term 2 LEADER]: Becoming Leader. State: Replica: c26a60fed59f474db4e0d0e1c38e9f0c, State: Running, Role: LEADER
I20260927 02:44:38.223461 13182 sys_catalog.cc:564] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:38.223579 13186 consensus_queue.cc:237] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } }
I20260927 02:44:38.223956 13188 sys_catalog.cc:455] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c26a60fed59f474db4e0d0e1c38e9f0c. Latest consensus state: current_term: 2 leader_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } }
I20260927 02:44:38.224030 13188 sys_catalog.cc:458] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:38.224236 13187 sys_catalog.cc:455] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26a60fed59f474db4e0d0e1c38e9f0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } }
I20260927 02:44:38.224296 13187 sys_catalog.cc:458] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:38.224473 13191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:38.225008 13191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:38.225201 13191 catalog_manager.cc:1259] Loaded cluster ID: acfd15b591dd4b88af56267d2570d2f9
I20260927 02:44:38.225279 13191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:38.225347 13182 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:44:38.228626 13191 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:38.229386 13191 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Loaded TSK: 0
I20260927 02:44:38.229678 13191 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:44:38.298823 13184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:38.299201 13184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:38.299304 13184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:38.301064 13184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:38.301167 13184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:38.301265 13184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:38.302947 13184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35341
--tserver_master_addrs=127.0.98.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:38.303190 13184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:38.303439 13184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:38.308861 13222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.308902 13184 server_base.cc:1034] running on GCE node
W20260927 02:44:38.309158 13219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.309224 13220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.309465 13184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:38.309719 13184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:38.310873 13184 hybrid_clock.cc:648] HybridClock initialized: now 1790477078310647 us; error 248 us; skew 500 ppm
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:38.313438 13184 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:38.313992 13184 webserver.cc:460] Webserver started at http://127.0.98.1:35341/ using document root <none> and password file <none>
I20260927 02:44:38.314235 13184 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:38.314287 13184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:38.344822 13184 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260927 02:44:38.347235 13230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.347393 13184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:38.347476 13184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "b09cc3bd10394e17b13e6de3e49f69be"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "bf586d942fa751bcad14a99f264c2385"
server_key_iv: "f4ad8f9f0b59c972cd6badd706e72854"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:38.347893 13184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:38.369866 13184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:38.371791 13184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:38.371932 13184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:38.372148 13184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:38.372509 13184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:38.372545 13184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.372604 13184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:38.372645 13184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.378875 13184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41035
I20260927 02:44:38.379526 13184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:38.380491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 13184
I20260927 02:44:38.380795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44567
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40511
--webserver_port=45861
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36711/kms with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:38.381184 13343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41035 every 8 connection(s)
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:38.409343 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.389085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40511 (local address 127.0.98.1:42593)
0927 02:44:38.396500 (+ 7415us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:38.396546 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:38.397087 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:38.397716 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:38.397720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:38.397974 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:38.398261 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.398268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.400472 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.400473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.401725 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.401728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.402648 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.402649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.402773 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.403372 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:38.403382 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:38.405386 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:38.407547 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.407554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.407556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.407841 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.408118 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.408119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.408119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.408240 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.408698 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:38.408704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:38.408864 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 02:44:38.408960 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 02:44:38.409126 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7363,"thread_start_us":89,"threads_started":1}
I20260927 02:44:38.410152 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.389143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42593 (local address 127.0.98.62:40511)
0927 02:44:38.397265 (+ 8122us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:38.397273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:38.397300 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:44:38.397414 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:38.397418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:38.397459 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:38.397583 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:38.398378 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.400322 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.401816 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.402447 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.402539 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.405536 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:38.405541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:38.405542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:38.405554 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:38.407450 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.407924 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.407927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.407928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.408031 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.408301 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.408302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.408303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.408600 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:38.408672 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:38.409788 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0927 02:44:38.409910 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8069,"thread_start_us":118,"threads_started":1}
I20260927 02:44:38.411219 13344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40511
I20260927 02:44:38.411311 13344 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.411485 13344 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
W20260927 02:44:38.412310 13344 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40511 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:44:38.412375 13344 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.412454 13344 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
I20260927 02:44:38.412825 13344 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.412890 13344 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
W20260927 02:44:38.413205 13344 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:44:38.509963 13346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:38.510262 13346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:38.510303 13346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:38.512113 13346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:38.512180 13346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:38.512221 13346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:38.514510 13346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45861
--tserver_master_addrs=127.0.98.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:38.514808 13346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:38.515108 13346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:38.520602 13356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.521374 13353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.521819 13346 server_base.cc:1034] running on GCE node
W20260927 02:44:38.520623 13354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.535107 13346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:38.535516 13346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:38.536717 13346 hybrid_clock.cc:648] HybridClock initialized: now 1790477078536701 us; error 47 us; skew 500 ppm
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:38.540181 13346 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:38.540818 13346 webserver.cc:460] Webserver started at http://127.0.98.2:45861/ using document root <none> and password file <none>
I20260927 02:44:38.541121 13346 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:38.541200 13346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:38.562041 13346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:38.562969 13364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.563235 13346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:38.563330 13346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "4d52ddba81ef4b15b2a644b8995520e3"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "cf93df5c2ca61c39b0ebc0ea26255729"
server_key_iv: "beb03e8f55450f7134fdb02a8344353a"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:38.563885 13346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:38.590680 13346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:38.592990 13346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:38.593163 13346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:38.593429 13346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:38.593863 13346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:38.593920 13346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.593952 13346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:38.593971 13346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.601287 13346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44567
I20260927 02:44:38.601866 13346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:38.602527 13477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44567 every 8 connection(s)
I20260927 02:44:38.608000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 13346
I20260927 02:44:38.608300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42109
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40511
--webserver_port=45261
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36711/kms with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:38.615345 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.602982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51151 (local address 127.0.98.62:40511)
0927 02:44:38.603036 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:38.603038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:38.603049 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:44:38.603662 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:38.603663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:38.603679 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:38.603747 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:38.604478 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.605230 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.606726 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.607039 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.607113 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.611805 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:38.611807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:38.611808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:38.611817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:38.613407 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.614056 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.614057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.614058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.614130 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.614447 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.614448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.614448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.614598 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:38.614672 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:38.615149 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 02:44:38.615245 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:44:38.616034 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.602693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40511 (local address 127.0.98.2:51151)
0927 02:44:38.602873 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:38.602918 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:38.603538 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:38.603829 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:38.603833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:38.604065 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:38.604364 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.604371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.605345 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.605347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.606638 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.606642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.607151 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.607152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.607301 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.610044 (+ 2743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:38.610055 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:38.611657 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:38.613505 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.613514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.613517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.613941 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.614216 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.614217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.614217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.614374 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.614714 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:38.614721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:38.614968 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 02:44:38.615609 (+ 641us) client_negotiation.cc:240] Negotiation successful
0927 02:44:38.615787 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260927 02:44:38.616597 13478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40511
I20260927 02:44:38.616711 13478 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.616919 13478 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
W20260927 02:44:38.617544 13478 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40511 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:44:38.617684 13478 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.617794 13478 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
I20260927 02:44:38.618120 13478 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.618206 13478 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
W20260927 02:44:38.618551 13478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:44:38.715929 13481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:38.716305 13481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:38.716404 13481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:38.718711 13481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:38.718832 13481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:38.718916 13481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:38.721284 13481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45261
--tserver_master_addrs=127.0.98.62:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:38.721853 13481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:38.722185 13481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:38.728610 13488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:38.728828 13487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.728964 13481 server_base.cc:1034] running on GCE node
W20260927 02:44:38.728642 13490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:38.729218 13481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:38.729483 13481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:38.730647 13481 hybrid_clock.cc:648] HybridClock initialized: now 1790477078730622 us; error 40 us; skew 500 ppm
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:38.733862 13481 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:38.734426 13481 webserver.cc:460] Webserver started at http://127.0.98.3:45261/ using document root <none> and password file <none>
I20260927 02:44:38.734709 13481 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:38.734812 13481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:38.753906 13481 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:38.754726 13498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.754930 13481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:38.755007 13481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "da44ab8f61eb4924abd060d53b261b73"
format_stamp: "Formatted at 2026-09-27 02:44:37 on dist-test-slave-44kr"
server_key: "a3b34ae83e9eebd1747150aadde23dc6"
server_key_iv: "67c3dc403c795d3fa82464c14b39ab88"
server_key_version: "kuduclusterkey@0"
I20260927 02:44:38.755431 13481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:38.803710 13481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:38.806126 13481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:38.806277 13481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:38.806524 13481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:38.806840 13481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:38.806874 13481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.806985 13481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:38.807036 13481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:38.814079 13481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42109
I20260927 02:44:38.814164 13611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42109 every 8 connection(s)
I20260927 02:44:38.814524 13481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:44:38 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:38.822678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 13481
I20260927 02:44:38.826808 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.816457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49029 (local address 127.0.98.62:40511)
0927 02:44:38.816504 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:38.816506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:38.816517 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:44:38.816545 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:38.816546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:38.816557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:38.816626 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:38.817425 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.818159 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.819740 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.820045 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.820092 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.823294 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:38.823296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:38.823296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:38.823305 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:38.825121 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.825645 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.825646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.825647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.825711 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.826012 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.826013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.826013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.826159 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:38.826234 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:38.826628 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 02:44:38.826714 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:44:38.827594 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.815289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40511 (local address 127.0.98.3:49029)
0927 02:44:38.815441 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:38.815487 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:38.815946 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:38.816727 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:38.816732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:38.816995 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:38.817300 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.817307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.818273 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.818274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.819625 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:38.819629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.820203 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.820204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:38.820355 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.821001 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:38.821012 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:38.822821 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:38.825225 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.825234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.825238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.825550 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.825783 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:38.825784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:38.825784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:38.825937 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:38.826245 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:38.826251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:38.826440 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 02:44:38.827182 (+ 742us) client_negotiation.cc:240] Negotiation successful
0927 02:44:38.827379 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260927 02:44:38.828049 13612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40511
I20260927 02:44:38.828166 13612 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.828420 13612 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
W20260927 02:44:38.829176 13612 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40511 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:44:38.829259 13612 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.829334 13612 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
I20260927 02:44:38.829646 13612 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:38.829733 13612 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
W20260927 02:44:38.830042 13612 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:44:38.834945 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:38.823122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.0.98.62:40511)
0927 02:44:38.826527 (+ 3405us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:38.826529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:38.827587 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 02:44:38.827617 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:38.827619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:38.827633 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:38.827717 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:38.828633 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.829859 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.830713 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:38.831018 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:38.831095 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:38.832295 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:38.832296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:38.832297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:38.832305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:38.833857 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.834193 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.834195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.834196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.834251 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:38.834412 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:38.834413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:38.834414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:38.834519 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:38.834716 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:38.834753 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 02:44:38.834842 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3380,"thread_start_us":64,"threads_started":1}
Sep 27 02:44:39 dist-test-slave-44kr krb5kdc[12150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:39.413391 13344 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:39.413558 13344 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
I20260927 02:44:39.414003 13146 ts_manager.cc:194] Registered new tserver with Master: b09cc3bd10394e17b13e6de3e49f69be (127.0.98.1:41035)
I20260927 02:44:39.415046 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42593
I20260927 02:44:39.618721 13478 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:39.618865 13478 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
I20260927 02:44:39.619123 13146 ts_manager.cc:194] Registered new tserver with Master: 4d52ddba81ef4b15b2a644b8995520e3 (127.0.98.2:44567)
I20260927 02:44:39.619510 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51151
Sep 27 02:44:39 dist-test-slave-44kr krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:44:39.830235 13612 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:39.830405 13612 heartbeater.cc:511] Master 127.0.98.62:40511 requested a full tablet report, sending...
I20260927 02:44:39.830724 13146 ts_manager.cc:194] Registered new tserver with Master: da44ab8f61eb4924abd060d53b261b73 (127.0.98.3:42109)
I20260927 02:44:39.831234 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49029
I20260927 02:44:39.840785 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:44:39.855459 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.841779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.0.98.62:40511)
0927 02:44:39.841908 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.841910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.847519 (+ 5609us) server_negotiation.cc:407] Connection header received
0927 02:44:39.847561 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.847562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.847579 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.847684 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:39.848432 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.849225 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.850272 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.850599 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.850672 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.852185 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:39.852187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:39.852188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:39.852196 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:39.854025 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:39.854446 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:39.854447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:39.854447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:39.854515 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:39.854732 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:39.854734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:39.854734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:39.854897 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:39.855003 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:39.855136 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:44:39.855228 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260927 02:44:39.857620 13146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:39.860030 13146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:39.868526 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.862956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41035 (local address 127.0.0.1:40584)
0927 02:44:39.863560 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.863609 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.863676 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.863937 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.863939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.863941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:39.864002 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.864004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.865325 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.865326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.866899 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.866902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.868163 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.868164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.868245 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.868275 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 02:44:39.868327 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:44:39.868440 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":78,"threads_started":1}
I20260927 02:44:39.868975 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.863363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.0.98.1:41035)
0927 02:44:39.863470 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.863473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.863749 (+ 276us) server_negotiation.cc:407] Connection header received
0927 02:44:39.863786 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.863788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.863803 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.863861 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:39.864085 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.865204 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.867005 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.868073 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.868737 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.868826 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:39.868855 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 02:44:39.868900 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":45,"threads_started":1}
I20260927 02:44:39.869680 13259 tablet_service.cc:1467] Processing CreateTablet for tablet 5116847feab24c2da51c9eecb6f32c6a (DEFAULT_TABLE table=test-table [id=23d0f5e2f2494b389d8a9af642979902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:39.870041 13259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5116847feab24c2da51c9eecb6f32c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:39.871507 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.863212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.0.98.2:44567)
0927 02:44:39.863496 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.863501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.864544 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 02:44:39.864756 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.864759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.864774 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.864821 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:39.865322 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.866298 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.870273 (+ 3975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.870971 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.871023 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.871092 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:39.871367 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 02:44:39.871432 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":67,"threads_started":1}
I20260927 02:44:39.871742 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.863264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.0.98.3:42109)
0927 02:44:39.863382 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.863387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.864699 (+ 1312us) server_negotiation.cc:407] Connection header received
0927 02:44:39.865071 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.865074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.865087 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.865135 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:39.867513 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.868576 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.870247 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.871294 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.871367 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.871439 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:39.871622 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 02:44:39.871663 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
I20260927 02:44:39.871985 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.863176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.0.1:41574)
0927 02:44:39.864648 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.864656 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.864996 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.865223 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.865223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.865225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:39.865254 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.865255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.868902 (+ 3647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.868903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.870160 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.870163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.871474 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.871475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.871535 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.871539 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:39.871862 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 02:44:39.871908 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1458,"thread_start_us":67,"threads_started":1}
I20260927 02:44:39.872294 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.862990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44567 (local address 127.0.0.1:40694)
0927 02:44:39.864470 (+ 1480us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.864480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.864571 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.864902 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.864903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.864904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:39.864945 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.864946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.866388 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.866389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.867408 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.867411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.871178 (+ 3767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.871179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.871242 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.871245 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:44:39.871911 (+ 666us) client_negotiation.cc:240] Negotiation successful
0927 02:44:39.871960 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":80,"threads_started":1}
I20260927 02:44:39.872687 13546 tablet_service.cc:1467] Processing CreateTablet for tablet 5116847feab24c2da51c9eecb6f32c6a (DEFAULT_TABLE table=test-table [id=23d0f5e2f2494b389d8a9af642979902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:39.873045 13546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5116847feab24c2da51c9eecb6f32c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:39.872984 13399 tablet_service.cc:1467] Processing CreateTablet for tablet 5116847feab24c2da51c9eecb6f32c6a (DEFAULT_TABLE table=test-table [id=23d0f5e2f2494b389d8a9af642979902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:39.873283 13399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5116847feab24c2da51c9eecb6f32c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:39.876519 13646 tablet_bootstrap.cc:492] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: Bootstrap starting.
I20260927 02:44:39.877679 13646 tablet_bootstrap.cc:654] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:39.877970 13646 log.cc:824] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:39.878458 13647 tablet_bootstrap.cc:492] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3: Bootstrap starting.
I20260927 02:44:39.879045 13647 tablet_bootstrap.cc:654] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:39.879314 13647 log.cc:824] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:39.880129 13647 tablet_bootstrap.cc:492] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3: No bootstrap required, opened a new log
I20260927 02:44:39.880201 13647 ts_tablet_manager.cc:1397] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:39.881028 13646 tablet_bootstrap.cc:492] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: No bootstrap required, opened a new log
I20260927 02:44:39.881102 13646 ts_tablet_manager.cc:1397] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 02:44:39.883648 13646 raft_consensus.cc:348] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:44:39.883839 13646 raft_consensus.cc:374] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:39.883869 13646 raft_consensus.cc:729] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da44ab8f61eb4924abd060d53b261b73, State: Initialized, Role: FOLLOWER
I20260927 02:44:39.884560 13645 tablet_bootstrap.cc:492] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be: Bootstrap starting.
I20260927 02:44:39.885270 13645 tablet_bootstrap.cc:654] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:39.885572 13645 log.cc:824] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:39.886222 13647 raft_consensus.cc:348] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:44:39.886392 13647 raft_consensus.cc:374] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:39.886421 13645 tablet_bootstrap.cc:492] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be: No bootstrap required, opened a new log
I20260927 02:44:39.886427 13647 raft_consensus.cc:729] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d52ddba81ef4b15b2a644b8995520e3, State: Initialized, Role: FOLLOWER
I20260927 02:44:39.886483 13645 ts_tablet_manager.cc:1397] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:39.886606 13646 consensus_queue.cc:260] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:44:39.886904 13646 ts_tablet_manager.cc:1428] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:44:39.887068 13612 heartbeater.cc:503] Master 127.0.98.62:40511 was elected leader, sending a full tablet report...
I20260927 02:44:39.887089 13647 consensus_queue.cc:260] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:44:39.887364 13647 ts_tablet_manager.cc:1428] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260927 02:44:39.887480 13478 heartbeater.cc:503] Master 127.0.98.62:40511 was elected leader, sending a full tablet report...
I20260927 02:44:39.888968 13645 raft_consensus.cc:348] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:44:39.889115 13645 raft_consensus.cc:374] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:39.889137 13645 raft_consensus.cc:729] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09cc3bd10394e17b13e6de3e49f69be, State: Initialized, Role: FOLLOWER
I20260927 02:44:39.889231 13645 consensus_queue.cc:260] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:44:39.889458 13645 ts_tablet_manager.cc:1428] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:39.889510 13344 heartbeater.cc:503] Master 127.0.98.62:40511 was elected leader, sending a full tablet report...
I20260927 02:44:39.914573 13653 raft_consensus.cc:482] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:39.914747 13653 raft_consensus.cc:504] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:44:39.915174 13653 leader_election.cc:288] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d52ddba81ef4b15b2a644b8995520e3 (127.0.98.2:44567), da44ab8f61eb4924abd060d53b261b73 (127.0.98.3:42109)
I20260927 02:44:39.919425 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.915621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45543 (local address 127.0.98.3:42109)
0927 02:44:39.915662 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.915663 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.915935 (+ 272us) server_negotiation.cc:407] Connection header received
0927 02:44:39.916018 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.916019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.916030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.916055 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:39.916272 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.916974 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.918509 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.919061 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.919105 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.919115 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:39.919313 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 02:44:39.919353 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:44:39.919479 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.915463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37857 (local address 127.0.98.2:44567)
0927 02:44:39.915509 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.915511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.915834 (+ 323us) server_negotiation.cc:407] Connection header received
0927 02:44:39.915869 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.915870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.915883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.915956 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:39.916223 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.916986 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.918465 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.919051 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.919101 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.919113 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:39.919361 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:44:39.919408 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:44:39.919744 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.915353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44567 (local address 127.0.98.1:37857)
0927 02:44:39.915745 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.915756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.915823 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.916013 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.916015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.916018 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:39.916114 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.916116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.917083 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.917083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.918359 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.918362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.919194 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.919195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.919268 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.919272 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:39.919593 (+ 321us) client_negotiation.cc:240] Negotiation successful
0927 02:44:39.919643 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":43,"thread_start_us":99,"threads_started":1}
I20260927 02:44:39.920145 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.915550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.98.1:45543)
0927 02:44:39.915890 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.915897 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:39.915956 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:39.916111 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:39.916112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:39.916113 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:39.916207 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.916209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.917064 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.917064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.918417 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:39.918420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.919174 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.919175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:39.919240 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.919244 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:39.919902 (+ 658us) client_negotiation.cc:240] Negotiation successful
0927 02:44:39.920047 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260927 02:44:39.920079 13414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5116847feab24c2da51c9eecb6f32c6a" candidate_uuid: "b09cc3bd10394e17b13e6de3e49f69be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d52ddba81ef4b15b2a644b8995520e3" is_pre_election: true
I20260927 02:44:39.920265 13414 raft_consensus.cc:2454] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09cc3bd10394e17b13e6de3e49f69be in term 0.
I20260927 02:44:39.920538 13234 leader_election.cc:302] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d52ddba81ef4b15b2a644b8995520e3, b09cc3bd10394e17b13e6de3e49f69be; no voters:
I20260927 02:44:39.920677 13653 raft_consensus.cc:2781] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:39.920624 13566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5116847feab24c2da51c9eecb6f32c6a" candidate_uuid: "b09cc3bd10394e17b13e6de3e49f69be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da44ab8f61eb4924abd060d53b261b73" is_pre_election: true
I20260927 02:44:39.920713 13653 raft_consensus.cc:482] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:39.920732 13653 raft_consensus.cc:3037] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:39.920771 13566 raft_consensus.cc:2454] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09cc3bd10394e17b13e6de3e49f69be in term 0.
I20260927 02:44:39.921548 13653 raft_consensus.cc:504] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:44:39.921680 13653 leader_election.cc:288] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [CANDIDATE]: Term 1 election: Requested vote from peers 4d52ddba81ef4b15b2a644b8995520e3 (127.0.98.2:44567), da44ab8f61eb4924abd060d53b261b73 (127.0.98.3:42109)
I20260927 02:44:39.921914 13566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5116847feab24c2da51c9eecb6f32c6a" candidate_uuid: "b09cc3bd10394e17b13e6de3e49f69be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da44ab8f61eb4924abd060d53b261b73"
I20260927 02:44:39.921983 13566 raft_consensus.cc:3037] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:39.922425 13414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5116847feab24c2da51c9eecb6f32c6a" candidate_uuid: "b09cc3bd10394e17b13e6de3e49f69be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d52ddba81ef4b15b2a644b8995520e3"
I20260927 02:44:39.922504 13414 raft_consensus.cc:3037] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:39.922787 13566 raft_consensus.cc:2454] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09cc3bd10394e17b13e6de3e49f69be in term 1.
I20260927 02:44:39.922958 13234 leader_election.cc:302] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09cc3bd10394e17b13e6de3e49f69be, da44ab8f61eb4924abd060d53b261b73; no voters:
I20260927 02:44:39.923029 13653 raft_consensus.cc:2781] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:39.923153 13653 raft_consensus.cc:686] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [term 1 LEADER]: Becoming Leader. State: Replica: b09cc3bd10394e17b13e6de3e49f69be, State: Running, Role: LEADER
I20260927 02:44:39.923234 13653 consensus_queue.cc:237] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:44:39.923379 13414 raft_consensus.cc:2454] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09cc3bd10394e17b13e6de3e49f69be in term 1.
I20260927 02:44:39.924124 13145 catalog_manager.cc:5495] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be reported cstate change: term changed from 0 to 1, leader changed from <none> to b09cc3bd10394e17b13e6de3e49f69be (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b09cc3bd10394e17b13e6de3e49f69be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09cc3bd10394e17b13e6de3e49f69be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:39.940431 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.937396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.0.98.1:41035)
0927 02:44:39.937436 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:39.937438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:39.937444 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:44:39.937481 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:39.937482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:39.937501 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:39.937558 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:39.937745 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.938409 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.939441 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:39.939762 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:39.939820 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:39.939997 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:39.940129 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:39.940208 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:39.940312 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 02:44:39.940334 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:44:39.942814 13566 raft_consensus.cc:1261] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [term 1 FOLLOWER]: Refusing update from remote peer b09cc3bd10394e17b13e6de3e49f69be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:39.943262 13656 consensus_queue.cc:1035] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [LEADER]: Connected to new peer: Peer: permanent_uuid: "da44ab8f61eb4924abd060d53b261b73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:39.943825 13414 raft_consensus.cc:1261] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [term 1 FOLLOWER]: Refusing update from remote peer b09cc3bd10394e17b13e6de3e49f69be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:39.944078 13653 consensus_queue.cc:1035] T 5116847feab24c2da51c9eecb6f32c6a P b09cc3bd10394e17b13e6de3e49f69be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d52ddba81ef4b15b2a644b8995520e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:39.946942 13660 mvcc.cc:205] Tried to move back new op lower bound from 7333794119443161088 to 7333794119366254592. Current Snapshot: MvccSnapshot[applied={T|T < 7333794119443161088}]
I20260927 02:44:39.949434 13145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:39.949522 13145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:39.950459 13145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c26a60fed59f474db4e0d0e1c38e9f0c: Sending DeleteTablet for 3 replicas of tablet 5116847feab24c2da51c9eecb6f32c6a
I20260927 02:44:39.950897 13399 tablet_service.cc:1514] Processing DeleteTablet for tablet 5116847feab24c2da51c9eecb6f32c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40694
I20260927 02:44:39.951112 13546 tablet_service.cc:1514] Processing DeleteTablet for tablet 5116847feab24c2da51c9eecb6f32c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41574
I20260927 02:44:39.951507 13259 tablet_service.cc:1514] Processing DeleteTablet for tablet 5116847feab24c2da51c9eecb6f32c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40584
I20260927 02:44:39.951778 13662 mvcc.cc:205] Tried to move back new op lower bound from 7333794119443161088 to 7333794119366254592. Current Snapshot: MvccSnapshot[applied={T|T < 7333794119443161088}]
I20260927 02:44:39.952152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 13184
I20260927 02:44:39.952978 13669 tablet_replica.cc:330] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: stopping tablet replica
I20260927 02:44:39.953162 13669 raft_consensus.cc:2229] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:39.953336 13669 raft_consensus.cc:2258] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:39.953614 13667 tablet_replica.cc:330] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3: stopping tablet replica
I20260927 02:44:39.953848 13667 raft_consensus.cc:2229] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:39.954268 13667 raft_consensus.cc:2258] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:39.954581 13669 ts_tablet_manager.cc:1905] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:39.956624 13669 ts_tablet_manager.cc:1918] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:39.956936 13667 ts_tablet_manager.cc:1905] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:39.958582 13667 ts_tablet_manager.cc:1918] T 5116847feab24c2da51c9eecb6f32c6a P 4d52ddba81ef4b15b2a644b8995520e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 02:44:39.958603 13130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41035 (error 108)
I20260927 02:44:39.958033 13669 log.cc:1196] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/5116847feab24c2da51c9eecb6f32c6a
I20260927 02:44:39.959143 13669 ts_tablet_manager.cc:1939] T 5116847feab24c2da51c9eecb6f32c6a P da44ab8f61eb4924abd060d53b261b73: Deleting consensus metadata
I20260927 02:44:39.959203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 13346
I20260927 02:44:39.960114 13133 catalog_manager.cc:4841] TS da44ab8f61eb4924abd060d53b261b73 (127.0.98.3:42109): tablet 5116847feab24c2da51c9eecb6f32c6a (table test-table [id=23d0f5e2f2494b389d8a9af642979902]) successfully deleted
I20260927 02:44:39.961632 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:39.961474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41035 (local address 127.0.0.1:40608)
0927 02:44:39.961546 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:39.961599 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:44:39.964890 13130 catalog_manager.cc:4584] TS b09cc3bd10394e17b13e6de3e49f69be (127.0.98.1:41035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5116847feab24c2da51c9eecb6f32c6a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41035: connect: Connection refused (error 111)
I20260927 02:44:39.965431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 13481
I20260927 02:44:39.971829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 13113
I20260927 02:44:39.976379 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:44:40.311098 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:44:40.311188 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:44:40.645682 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:44:40.645 UTC [12180] LOG: received smart shutdown request
2026-09-27 02:44:40.649 UTC [12180] LOG: background worker "logical replication launcher" (PID 12189) exited with exit code 1
2026-09-27 02:44:40.650 UTC [12184] LOG: shutting down
2026-09-27 02:44:40.841 UTC [12180] LOG: database system is shut down
2026-09-27T02:44:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48519 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:40 dist-test-slave-44kr krb5kdc[13681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:40 dist-test-slave-44kr krb5kdc[13681](info): set up 2 sockets
Sep 27 02:44:40 dist-test-slave-44kr krb5kdc[13681](info): commencing operation
krb5kdc: starting...
W20260927 02:44:41.892446 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:41 dist-test-slave-44kr krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:42.012686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38381,127.0.98.61:37251,127.0.98.60:41435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:42.093231 13694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:42.093461 13694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:42.093484 13694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:42.094743 13694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:42.094816 13694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:42.094831 13694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:42.094842 13694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:42.094854 13694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:42.096549 13694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38381,127.0.98.61:37251,127.0.98.60:41435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:42.096796 13694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:42.097030 13694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:42.102473 13699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:42.102491 13700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:42.102492 13702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:42.102735 13694 server_base.cc:1034] running on GCE node
I20260927 02:44:42.102999 13694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:42.103300 13694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:42.104475 13694 hybrid_clock.cc:648] HybridClock initialized: now 1790477082104433 us; error 58 us; skew 500 ppm
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:42.106931 13694 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:42.107450 13694 webserver.cc:460] Webserver started at http://127.0.98.62:42259/ using document root <none> and password file <none>
I20260927 02:44:42.107676 13694 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:42.107725 13694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:42.107849 13694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:42.108958 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "9a8a2bf5b0794855bbddf3f36583f676"
format_stamp: "Formatted at 2026-09-27 02:44:42 on dist-test-slave-44kr"
server_key: "07ea80a731d2eb5952f7781d9ea42bdf"
server_key_iv: "d6e81d267db63aac6704f0e0635206a8"
server_key_version: "encryptionkey@0"
I20260927 02:44:42.109287 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a8a2bf5b0794855bbddf3f36583f676"
format_stamp: "Formatted at 2026-09-27 02:44:42 on dist-test-slave-44kr"
server_key: "07ea80a731d2eb5952f7781d9ea42bdf"
server_key_iv: "d6e81d267db63aac6704f0e0635206a8"
server_key_version: "encryptionkey@0"
I20260927 02:44:42.110515 13694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:44:42.111367 13709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:42.111531 13694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:42.111613 13694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "9a8a2bf5b0794855bbddf3f36583f676"
format_stamp: "Formatted at 2026-09-27 02:44:42 on dist-test-slave-44kr"
server_key: "07ea80a731d2eb5952f7781d9ea42bdf"
server_key_iv: "d6e81d267db63aac6704f0e0635206a8"
server_key_version: "encryptionkey@0"
I20260927 02:44:42.111673 13694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:42.122347 13694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:42.124619 13694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:42.124776 13694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:42.128950 13694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38381
I20260927 02:44:42.128984 13761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38381 every 8 connection(s)
I20260927 02:44:42.129323 13694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:42.130225 13762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:42.131853 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:44:42.137763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 13694
I20260927 02:44:42.137895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-0/wal/instance
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:42.138096 392 external_mini_cluster.cc:1428] Setting key 2dc0aa8d1bf8c17378dd5237b48e01f5
I20260927 02:44:42.142306 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.132810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.0.98.62:38381)
0927 02:44:42.132931 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.132935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.133004 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:44:42.133585 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.133587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.133608 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.133682 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.134216 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.135035 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.135875 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.136203 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.136296 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.138780 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.138789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.138790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.138797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.140462 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.140919 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.140923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.140923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.141014 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.141320 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.141321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.141322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.141599 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.141683 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.141959 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.142124 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":97,"thread_start_us":44,"threads_started":1}
I20260927 02:44:42.142318 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.132632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38381 (local address 127.0.0.1:34436)
0927 02:44:42.132851 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.132914 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.133429 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.133695 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.133698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.133964 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.134130 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.134135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.135124 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.135124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.135799 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.135801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.136273 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.136274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.136338 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.136876 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.136885 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.138667 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:42.140553 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.140561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.140564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.140820 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.141128 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.141129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.141129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.141234 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.141673 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:42.141679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:42.141762 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 02:44:42.141872 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 02:44:42.142062 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260927 02:44:42.143229 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:44:42.143615 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.143465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37251 (local address 127.0.0.1:58786)
0927 02:44:42.143522 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.143585 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 02:44:42.143787 13713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
I20260927 02:44:42.144277 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.144177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37251 (local address 127.0.0.1:58794)
0927 02:44:42.144224 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.144255 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 02:44:42.144434 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37251: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:44:42.168217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37251
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38381,127.0.98.61:37251,127.0.98.60:41435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:42.193692 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } attempt: 1
I20260927 02:44:42.194312 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.194135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37251 (local address 127.0.0.1:58806)
0927 02:44:42.194227 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.194277 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 02:44:42.194772 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.194621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37251 (local address 127.0.0.1:58808)
0927 02:44:42.194668 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.194746 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:44:42.194919 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37251: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
W20260927 02:44:42.249778 13769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:42.250008 13769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:42.250031 13769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:42.251219 13769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:42.251261 13769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:42.251274 13769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:42.251284 13769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:42.251295 13769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:42.252890 13769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38381,127.0.98.61:37251,127.0.98.60:41435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:42.253110 13769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:42.253309 13769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:44:42.258427 13769 server_base.cc:1034] running on GCE node
W20260927 02:44:42.258371 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:42.258354 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:42.258350 13777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:42.258881 13769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:42.259195 13769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:42.260339 13769 hybrid_clock.cc:648] HybridClock initialized: now 1790477082260324 us; error 29 us; skew 500 ppm
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:42.263013 13769 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:44:42.263082 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } attempt: 2
I20260927 02:44:42.263455 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.263344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37251 (local address 127.0.0.1:58810)
0927 02:44:42.263396 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.263424 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:44:42.263579 13769 webserver.cc:460] Webserver started at http://127.0.98.61:37787/ using document root <none> and password file <none>
I20260927 02:44:42.263777 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.263672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37251 (local address 127.0.0.1:58812)
0927 02:44:42.263712 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.263748 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 02:44:42.263844 13769 fs_manager.cc:359] Metadata directory not provided
W20260927 02:44:42.263897 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37251: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37251: connect: Connection refused (error 111)
I20260927 02:44:42.263907 13769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:42.264030 13769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:42.265113 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/data/instance:
uuid: "e2d4d663721a4d6eb1406b9424ea83a5"
format_stamp: "Formatted at 2026-09-27 02:44:42 on dist-test-slave-44kr"
server_key: "0369de567b77be8352490b4c323dc1a3"
server_key_iv: "a4acc877d0056a60e41ecd43d5f807a5"
server_key_version: "encryptionkey@0"
I20260927 02:44:42.265416 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/wal/instance:
uuid: "e2d4d663721a4d6eb1406b9424ea83a5"
format_stamp: "Formatted at 2026-09-27 02:44:42 on dist-test-slave-44kr"
server_key: "0369de567b77be8352490b4c323dc1a3"
server_key_iv: "a4acc877d0056a60e41ecd43d5f807a5"
server_key_version: "encryptionkey@0"
I20260927 02:44:42.266606 13769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 02:44:42.267369 13784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:42.267534 13769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:42.267617 13769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/wal
uuid: "e2d4d663721a4d6eb1406b9424ea83a5"
format_stamp: "Formatted at 2026-09-27 02:44:42 on dist-test-slave-44kr"
server_key: "0369de567b77be8352490b4c323dc1a3"
server_key_iv: "a4acc877d0056a60e41ecd43d5f807a5"
server_key_version: "encryptionkey@0"
I20260927 02:44:42.267674 13769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:42.283803 13769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:42.286288 13769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:42.286420 13769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:42.290210 13769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37251
I20260927 02:44:42.290252 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37251 every 8 connection(s)
I20260927 02:44:42.290565 13769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:44:42.291487 13837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:42.293012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 13769
I20260927 02:44:42.293115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-1/wal/instance
I20260927 02:44:42.293304 392 external_mini_cluster.cc:1428] Setting key 2943f47c515d94a9786321661817eb89
I20260927 02:44:42.293134 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 02:44:42.302865 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.293976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.0.98.62:38381)
0927 02:44:42.294042 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.294045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.294251 (+ 206us) server_negotiation.cc:407] Connection header received
0927 02:44:42.294834 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.294835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.294848 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.294917 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.295593 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.296298 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.297402 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.297719 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.297802 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.299936 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.299938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.299938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.299946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.301358 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.301750 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.301752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.301752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.301810 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.302088 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.302089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.302090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.302306 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.302400 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.302715 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.302793 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:44:42.303010 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.293931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38381 (local address 127.0.0.1:34452)
0927 02:44:42.294103 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.294157 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.294718 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.295018 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.295022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.295288 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.295497 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.295501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.296438 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.296440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.297309 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.297312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.297822 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.297823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.297945 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.298497 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.298513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.299833 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:42.301437 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.301444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.301446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.301662 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.301895 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.301897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.301897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.302006 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.302402 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:42.302408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:42.302539 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 02:44:42.302669 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 02:44:42.302818 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 02:44:42.303558 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:44:42.311874 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.303931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.0.98.61:37251)
0927 02:44:42.304071 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.304075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.304090 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:44:42.304104 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.304106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.304115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.304173 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.304513 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.305543 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.306245 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.306679 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.306751 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.308907 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.308919 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.308920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.308928 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.310724 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.311079 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.311083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.311083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.311144 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.311314 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.311314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.311315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.311516 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.311597 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.311693 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.311787 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260927 02:44:42.311946 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.303869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37251 (local address 127.0.0.1:58816)
0927 02:44:42.303918 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.303930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.303973 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.304192 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.304193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.304330 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.304416 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.304418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.305664 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.305665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.306166 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.306167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.306756 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.306757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.306824 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.307373 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.307377 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.308814 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:42.310828 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.310829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.310830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.311002 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.311211 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.311211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.311211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.311257 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.311581 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:42.311581 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:42.311618 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:44:42.311662 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:44:42.311841 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":84}
I20260927 02:44:42.312625 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:44:42.312948 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.312846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47346)
0927 02:44:42.312886 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.312930 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:44:42.313076 13785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
I20260927 02:44:42.313483 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.313389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47348)
0927 02:44:42.313435 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.313462 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 02:44:42.313575 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41435: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:44:42.320014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41435
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38381,127.0.98.61:37251,127.0.98.60:41435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:42.355060 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } attempt: 3
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:44:42.362820 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.355571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.0.98.61:37251)
0927 02:44:42.355664 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.355666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.355681 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:44:42.355712 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.355713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.355727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.355798 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.356172 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.356991 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.357708 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.358021 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.358086 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.360275 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.360276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.360277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.360284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.361660 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.362037 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.362039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.362039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.362099 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.362293 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.362293 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.362294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.362404 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.362457 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.362638 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.362716 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 02:44:42.362820 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } attempt: 1
I20260927 02:44:42.363502 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.363203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47362)
0927 02:44:42.363434 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.363470 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260927 02:44:42.363587 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.355521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37251 (local address 127.0.0.1:58822)
0927 02:44:42.355571 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.355584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.355645 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.355831 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.355832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.355985 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.356051 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.356053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.357091 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.357092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.357628 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.357631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.358107 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.358108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.358187 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.358664 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.358668 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.360173 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:42.361736 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.361737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.361738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.361952 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.362162 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.362163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.362163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.362224 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.362458 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:42.362459 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:42.362511 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 02:44:42.362575 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:44:42.362676 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 02:44:42.363839 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.363753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47376)
0927 02:44:42.363791 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.363813 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:44:42.363951 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41435: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
I20260927 02:44:42.364046 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:44:42.364378 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.364278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47384)
0927 02:44:42.364322 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.364355 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:44:42.364706 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.364632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47394)
0927 02:44:42.364667 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.364690 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:44:42.364794 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41435: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
I20260927 02:44:42.395951 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } attempt: 1
I20260927 02:44:42.396484 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.396313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47398)
0927 02:44:42.396374 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.396447 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 02:44:42.396874 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.396779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47410)
0927 02:44:42.396819 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.396852 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:44:42.396962 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41435: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
W20260927 02:44:42.402841 13844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:42.403043 13844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:42.403064 13844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:42.404178 13844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:42.404217 13844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:42.404229 13844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:42.404241 13844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:42.404251 13844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:42.405776 13844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38381,127.0.98.61:37251,127.0.98.60:41435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:42.406003 13844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:42.406215 13844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:42.411302 13849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:42.411288 13852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:42.411288 13850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:42.411558 13844 server_base.cc:1034] running on GCE node
I20260927 02:44:42.411852 13844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:42.412154 13844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:42.413322 13844 hybrid_clock.cc:648] HybridClock initialized: now 1790477082413301 us; error 37 us; skew 500 ppm
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:42.416235 13844 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:44:42.416771 13844 webserver.cc:460] Webserver started at http://127.0.98.60:37351/ using document root <none> and password file <none>
I20260927 02:44:42.417047 13844 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:42.417112 13844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:42.417255 13844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:42.418393 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/data/instance:
uuid: "d41d157625244176b56bda196ac6f737"
format_stamp: "Formatted at 2026-09-27 02:44:42 on dist-test-slave-44kr"
server_key: "2b1eaa828b56bb9c15d09b249a742d86"
server_key_iv: "1df4be6b836d52fe5ad54244d32f6f03"
server_key_version: "encryptionkey@0"
I20260927 02:44:42.418741 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/wal/instance:
uuid: "d41d157625244176b56bda196ac6f737"
format_stamp: "Formatted at 2026-09-27 02:44:42 on dist-test-slave-44kr"
server_key: "2b1eaa828b56bb9c15d09b249a742d86"
server_key_iv: "1df4be6b836d52fe5ad54244d32f6f03"
server_key_version: "encryptionkey@0"
I20260927 02:44:42.420028 13844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:44:42.420781 13859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:42.420933 13844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:42.421020 13844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/wal
uuid: "d41d157625244176b56bda196ac6f737"
format_stamp: "Formatted at 2026-09-27 02:44:42 on dist-test-slave-44kr"
server_key: "2b1eaa828b56bb9c15d09b249a742d86"
server_key_iv: "1df4be6b836d52fe5ad54244d32f6f03"
server_key_version: "encryptionkey@0"
I20260927 02:44:42.421082 13844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:42.432135 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } attempt: 2
I20260927 02:44:42.432662 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.432513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47426)
0927 02:44:42.432573 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.432620 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:44:42.433135 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.432969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47440)
0927 02:44:42.433026 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.433056 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 02:44:42.433257 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41435: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41435: connect: Connection refused (error 111)
I20260927 02:44:42.443142 13844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:42.445362 13844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:42.445509 13844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:42.449930 13844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41435
I20260927 02:44:42.449973 13911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41435 every 8 connection(s)
I20260927 02:44:42.450305 13844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:44:42.451174 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:42.452883 13912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:44:42.455194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 13844
I20260927 02:44:42.455298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476929252373-392-0/minicluster-data/master-2/wal/instance
I20260927 02:44:42.455494 392 external_mini_cluster.cc:1428] Setting key 013480a8a17c91b63ffab10eb05e07ac
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:44:42.462970 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.453580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.0.98.62:38381)
0927 02:44:42.453631 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.453633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.453875 (+ 242us) server_negotiation.cc:407] Connection header received
0927 02:44:42.454387 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.454389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.454403 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.454484 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.455161 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.455883 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.457224 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.457483 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.457566 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.460096 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.460097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.460098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.460124 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.461555 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.461946 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.461948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.461948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.462009 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.462293 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.462294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.462294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.462429 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.462480 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.462813 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.462893 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:44:42.463179 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.453556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38381 (local address 127.0.0.1:34458)
0927 02:44:42.453756 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.453799 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.454261 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.454545 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.454549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.454845 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.455048 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.455053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.455976 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.455977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.457115 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.457118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.457590 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.457591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.457711 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.458207 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.458216 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.460016 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:42.461637 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.461645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.461647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.461875 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.462064 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.462065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.462066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.462193 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.462487 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:42.462495 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:42.462663 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 02:44:42.462776 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 02:44:42.462943 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260927 02:44:42.463762 13912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:44:42.464887 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.455878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.0.98.60:41435)
0927 02:44:42.456053 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.456058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.456631 (+ 573us) server_negotiation.cc:407] Connection header received
0927 02:44:42.456656 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.456658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.456689 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.456756 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.457454 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.458241 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.459171 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.459634 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.459690 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.461994 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.462000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.462000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.462006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.463700 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.464062 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.464065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.464066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.464126 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.464312 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.464313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.464314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.464512 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.464672 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.464715 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.464788 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:42.470583 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.464015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37251 (local address 127.0.0.1:58824)
0927 02:44:42.464056 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.464066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.464095 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.464241 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.464242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.464409 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.464447 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.464450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.465268 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.465269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.465862 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.465865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.466291 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.466292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.466351 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.466653 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.466657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.468013 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:42.469450 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.469451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.469451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.469736 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.469931 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.469932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.469932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.469995 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.470244 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:42.470245 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:42.470354 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 02:44:42.470425 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:44:42.470510 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:44:42.470764 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.464074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.0.98.61:37251)
0927 02:44:42.464125 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.464128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.464136 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:44:42.464158 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.464160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.464170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.464221 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.464514 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.465200 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.465934 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.466215 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.466274 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.468088 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.468089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.468089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.468097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.469364 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.469813 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.469814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.469814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.469877 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.470049 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.470050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.470050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.470162 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.470219 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.470463 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.470548 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:44:42.471101 13912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:44:42.472110 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } attempt: 2
I20260927 02:44:42.473189 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.465585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.0.98.62:38381)
0927 02:44:42.465620 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.465622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.465645 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:44:42.465663 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.465664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.465672 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.465729 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.466168 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.466803 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.467625 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.467843 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.467909 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.470206 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.470207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.470208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.470214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.472083 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.472446 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.472447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.472448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.472500 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.472662 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.472663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.472663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.472790 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.472985 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.473022 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.473096 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:44:42.479344 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.472339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47474)
0927 02:44:42.472412 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.472421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.472456 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.472684 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.472685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.472868 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.472907 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.472909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.473846 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.473847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.474549 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.474552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.475026 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.475027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.475087 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.475443 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.475447 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.476786 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:42.478299 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.478300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.478301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.478473 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.478684 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.478685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.478685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.478748 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.479054 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:42.479055 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:42.479098 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:44:42.479157 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:44:42.479248 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:44:42.479352 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.471427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47462)
0927 02:44:42.471461 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.471471 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.471509 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.471615 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.471617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.471744 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.471782 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.471784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.472771 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.472772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.473457 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.473459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.473900 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.473901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.473943 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.474316 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.474320 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.475536 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:42.478303 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.478303 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.478304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.478482 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.478686 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.478686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.478687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.478736 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.479070 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:42.479071 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:42.479105 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:44:42.479157 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:44:42.479237 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:44:42.479606 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.471461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.0.98.60:41435)
0927 02:44:42.471499 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.471501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.471508 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:44:42.471557 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.471558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.471567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.471613 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.471900 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.472699 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.473515 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.473806 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.473860 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.475602 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.475603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.475604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.475610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.477645 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.478551 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.478552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.478553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.478632 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.478800 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.478801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.478802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.478990 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.479049 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.479474 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.479535 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:44:42.480002 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.472429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.0.98.60:41435)
0927 02:44:42.472571 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.472573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.472579 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:44:42.472595 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.472595 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.472604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.472975 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.473016 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.473753 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.474640 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.474945 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.475010 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.476886 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.476888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.476888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.476894 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.478222 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.478538 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.478540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.478540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.478608 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.478817 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.478818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.478819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.478992 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.479047 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.479186 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.479290 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260927 02:44:42.480536 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.473699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.0.98.61:37251)
0927 02:44:42.473730 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.473732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.473740 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:44:42.473767 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.473768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.473776 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.473819 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.474292 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.474733 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.475533 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.475775 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.475830 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.477958 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.477960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.477960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.477967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.479499 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.479817 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.479818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.479818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.479864 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.480048 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.480049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.480049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.480182 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.480318 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.480355 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.480457 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:44:42.481253 13912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737: Bootstrap starting.
I20260927 02:44:42.482048 13912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:42.482342 13912 log.cc:824] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:42.482746 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:44:42.482928 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676: Bootstrap starting.
I20260927 02:44:42.483474 13762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:42.483745 13762 log.cc:824] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:42.484145 13912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737: No bootstrap required, opened a new log
I20260927 02:44:42.484864 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676: No bootstrap required, opened a new log
I20260927 02:44:42.485801 13912 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } }
I20260927 02:44:42.485921 13912 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:42.485952 13912 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41d157625244176b56bda196ac6f737, State: Initialized, Role: FOLLOWER
I20260927 02:44:42.486085 13912 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } }
I20260927 02:44:42.486423 13912 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:42.486742 13762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } }
I20260927 02:44:42.486861 13762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:42.486896 13762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8a2bf5b0794855bbddf3f36583f676, State: Initialized, Role: FOLLOWER
I20260927 02:44:42.486994 13762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } }
I20260927 02:44:42.487251 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.487278 13762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:42.487973 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.488057 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:44:42.488502 13939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:44:42.488586 13939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 02:44:42.489257 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:44:42.489450 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:44:42.488261 13844 master_runner.cc:186] Error getting master registration for 127.0.98.62:38381: OK, instance_id { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" instance_seqno: 1790477082127781 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:44:42.490578 13844 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:44:42.489907 13694 master_runner.cc:186] Error getting master registration for 127.0.98.61:37251: OK, instance_id { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" instance_seqno: 1790477082289169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:44:42.493407 13694 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:44:42.487413 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:42.494199 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.484078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.0.98.60:41435)
0927 02:44:42.484108 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.484110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.484176 (+ 66us) server_negotiation.cc:407] Connection header received
0927 02:44:42.484247 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.484248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.484259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.484303 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.484990 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.485715 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.486689 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.486964 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.487008 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.491324 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.491325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.491326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.491333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.492798 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.493165 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.493167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.493167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.493220 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.493420 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.493421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.493421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.493544 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.493652 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.494030 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.494109 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20260927 02:44:42.497798 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.483973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.0.98.61:37251)
0927 02:44:42.484029 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.484031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.488215 (+ 4184us) server_negotiation.cc:407] Connection header received
0927 02:44:42.488325 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.488326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.488338 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.488367 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.489042 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.489718 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.490661 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.490921 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.490959 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.495004 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.495005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.495006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.495014 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.496585 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.496960 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.496961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.496962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.497018 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.497202 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.497202 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.497203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.497339 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.497433 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.497624 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.497703 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:44:42.501340 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.484615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.0.98.62:38381)
0927 02:44:42.484650 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.484652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.493685 (+ 9033us) server_negotiation.cc:407] Connection header received
0927 02:44:42.493795 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.493796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.493807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.493834 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.494462 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.495522 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.496867 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.497151 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.497190 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.498530 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.498531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.498532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.498539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.500095 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.500504 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.500505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.500506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.500561 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.500765 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.500766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.500766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.500891 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.500978 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.501171 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.501248 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:44:42.524427 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } attempt: 3
Sep 27 02:44:42 dist-test-slave-44kr krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:44:42.533082 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.524731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41435 (local address 127.0.0.1:47494)
0927 02:44:42.524795 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:42.524810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:42.524861 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:42.525011 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:42.525012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:42.525210 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:42.525265 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.525268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.526205 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.526206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.527040 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:42.527042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.527527 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.527528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:42.527587 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.528096 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:42.528102 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:42.529731 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:42.531883 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.531884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.531884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.532117 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.532326 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:42.532327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:42.532327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:42.532417 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:42.532734 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:42.532735 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:42.532791 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 02:44:42.532863 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:44:42.532971 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 02:44:42.533092 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:42.524824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.98.60:41435)
0927 02:44:42.524863 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:42.524865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:42.524874 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:44:42.524929 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:42.524930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:42.524940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:42.524997 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:42.525344 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.526119 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.527130 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:42.527451 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:42.527521 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:42.529844 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:42.529846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:42.529846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:42.529855 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:42.531785 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.532195 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.532196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.532196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.532260 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:42.532468 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:42.532469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:42.532470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:42.532661 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:42.532734 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:42.532910 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:44:42.532998 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:44:42.535194 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5: Bootstrap starting.
I20260927 02:44:42.535995 13837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:42.536334 13837 log.cc:824] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:42.537333 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5: No bootstrap required, opened a new log
I20260927 02:44:42.539368 13837 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } }
I20260927 02:44:42.539530 13837 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:42.539564 13837 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d4d663721a4d6eb1406b9424ea83a5, State: Initialized, Role: FOLLOWER
I20260927 02:44:42.539732 13837 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } }
I20260927 02:44:42.540741 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.540807 13837 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:42.540838 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:44:42.542747 13965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:44:42.542804 13965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:44:42.544988 13769 master_runner.cc:263] No leader master found from master e2d4d663721a4d6eb1406b9424ea83a5
I20260927 02:44:42.545040 13769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:44:42.553362 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:42.553483 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } }
I20260927 02:44:42.553758 13954 leader_election.cc:288] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a8a2bf5b0794855bbddf3f36583f676 (127.0.98.62:38381), d41d157625244176b56bda196ac6f737 (127.0.98.60:41435)
I20260927 02:44:42.554044 13886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d41d157625244176b56bda196ac6f737" is_pre_election: true
I20260927 02:44:42.554200 13886 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d4d663721a4d6eb1406b9424ea83a5 in term 0.
I20260927 02:44:42.554384 13785 leader_election.cc:302] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d41d157625244176b56bda196ac6f737, e2d4d663721a4d6eb1406b9424ea83a5; no voters:
I20260927 02:44:42.554829 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:42.554898 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:42.554921 13954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:42.556162 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } }
I20260927 02:44:42.556295 13954 leader_election.cc:288] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a8a2bf5b0794855bbddf3f36583f676 (127.0.98.62:38381), d41d157625244176b56bda196ac6f737 (127.0.98.60:41435)
I20260927 02:44:42.556731 13886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d41d157625244176b56bda196ac6f737"
I20260927 02:44:42.556814 13886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:42.557577 13886 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d4d663721a4d6eb1406b9424ea83a5 in term 1.
I20260927 02:44:42.557883 13785 leader_election.cc:302] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d41d157625244176b56bda196ac6f737, e2d4d663721a4d6eb1406b9424ea83a5; no voters:
I20260927 02:44:42.558166 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8a2bf5b0794855bbddf3f36583f676"
I20260927 02:44:42.558298 13735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:42.559098 13735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d4d663721a4d6eb1406b9424ea83a5 in term 1.
I20260927 02:44:42.559226 13736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8a2bf5b0794855bbddf3f36583f676" is_pre_election: true
I20260927 02:44:42.559293 13736 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e2d4d663721a4d6eb1406b9424ea83a5 in term 1. Re-sending same reply.
I20260927 02:44:42.561529 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:42.561717 13954 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [term 1 LEADER]: Becoming Leader. State: Replica: e2d4d663721a4d6eb1406b9424ea83a5, State: Running, Role: LEADER
I20260927 02:44:42.561832 13954 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } }
I20260927 02:44:42.562251 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d4d663721a4d6eb1406b9424ea83a5. Latest consensus state: current_term: 1 leader_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.562325 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:42.563499 13969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:42.563781 13969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:42.565508 13886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [term 1 FOLLOWER]: Refusing update from remote peer e2d4d663721a4d6eb1406b9424ea83a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:42.565939 13967 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:42.567302 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d4d663721a4d6eb1406b9424ea83a5. Latest consensus state: current_term: 1 leader_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.567405 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:42.567625 13736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [term 1 FOLLOWER]: Refusing update from remote peer e2d4d663721a4d6eb1406b9424ea83a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:42.568009 13967 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:42.569274 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d4d663721a4d6eb1406b9424ea83a5. Latest consensus state: current_term: 1 leader_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.569382 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:42.571789 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.571887 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:42.573097 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.573186 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:42.573393 13970 mvcc.cc:205] Tried to move back new op lower bound from 7333794130185822208 to 7333794130173935616. Current Snapshot: MvccSnapshot[applied={T|T < 7333794130185822208}]
I20260927 02:44:42.573542 13971 mvcc.cc:205] Tried to move back new op lower bound from 7333794130185822208 to 7333794130173935616. Current Snapshot: MvccSnapshot[applied={T|T < 7333794130185822208}]
I20260927 02:44:42.573982 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.574060 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:42.574527 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8a2bf5b0794855bbddf3f36583f676" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } peers { permanent_uuid: "e2d4d663721a4d6eb1406b9424ea83a5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37251 } } peers { permanent_uuid: "d41d157625244176b56bda196ac6f737" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41435 } } }
I20260927 02:44:42.574611 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:44:42.574932 13973 mvcc.cc:205] Tried to move back new op lower bound from 7333794130185822208 to 7333794130173935616. Current Snapshot: MvccSnapshot[applied={T|T < 7333794130185822208}]
I20260927 02:44:42.575582 13969 catalog_manager.cc:1347] Generated new cluster ID: 484b41ee9a094f3d99f4bed45a2c7064
I20260927 02:44:42.575630 13969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:42.601464 13969 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:42.602308 13969 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:42.613622 13969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2d4d663721a4d6eb1406b9424ea83a5: Generated new TSK 0
I20260927 02:44:42.613977 13969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:44:43.489104 13939 catalog_manager.cc:1259] Loaded cluster ID: 484b41ee9a094f3d99f4bed45a2c7064
I20260927 02:44:43.489176 13939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737: loading cluster ID for follower catalog manager: success
I20260927 02:44:43.489850 13951 catalog_manager.cc:1259] Loaded cluster ID: 484b41ee9a094f3d99f4bed45a2c7064
I20260927 02:44:43.489907 13951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676: loading cluster ID for follower catalog manager: success
I20260927 02:44:43.490175 13939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737: acquiring CA information for follower catalog manager: success
I20260927 02:44:43.490494 13939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d41d157625244176b56bda196ac6f737: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:44:43.490778 13951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676: acquiring CA information for follower catalog manager: success
I20260927 02:44:43.491034 13951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a8a2bf5b0794855bbddf3f36583f676: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:44:43.677625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 13694
I20260927 02:44:43.682173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 13769
I20260927 02:44:43.686648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 13844
2026-09-27T02:44:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2757 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:43 dist-test-slave-44kr krb5kdc[13985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:43 dist-test-slave-44kr krb5kdc[13985](info): set up 2 sockets
Sep 27 02:44:43 dist-test-slave-44kr krb5kdc[13985](info): commencing operation
krb5kdc: starting...
W20260927 02:44:44.640213 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:44 dist-test-slave-44kr krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:44.759654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:44.839885 13998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:44.840126 13998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:44.840148 13998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:44.841373 13998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:44.841426 13998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:44.841441 13998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:44.841454 13998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:44.841466 13998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:44.843057 13998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:44.843282 13998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:44.843463 13998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:44:44.848694 13998 server_base.cc:1034] running on GCE node
W20260927 02:44:44.848657 14006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:44.848675 14004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:44.848665 14003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:44.849203 13998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:44.849522 13998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:44.850687 13998 hybrid_clock.cc:648] HybridClock initialized: now 1790477084850672 us; error 43 us; skew 500 ppm
Sep 27 02:44:44 dist-test-slave-44kr krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:44.853359 13998 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:44.853837 13998 webserver.cc:460] Webserver started at http://127.0.98.62:45239/ using document root <none> and password file <none>
I20260927 02:44:44.854071 13998 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:44.854130 13998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:44.854259 13998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:44.855353 13998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "62478bbefd094758bd3128340e9adddc"
format_stamp: "Formatted at 2026-09-27 02:44:44 on dist-test-slave-44kr"
server_key: "96dabb8fe8d7713e3cd41cf83ca3e656"
server_key_iv: "2851c61c08f755c280767634e3f1b983"
server_key_version: "encryptionkey@0"
I20260927 02:44:44.855684 13998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "62478bbefd094758bd3128340e9adddc"
format_stamp: "Formatted at 2026-09-27 02:44:44 on dist-test-slave-44kr"
server_key: "96dabb8fe8d7713e3cd41cf83ca3e656"
server_key_iv: "2851c61c08f755c280767634e3f1b983"
server_key_version: "encryptionkey@0"
I20260927 02:44:44.857079 13998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:44:44.857926 14013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:44.858124 13998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:44.858240 13998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "62478bbefd094758bd3128340e9adddc"
format_stamp: "Formatted at 2026-09-27 02:44:44 on dist-test-slave-44kr"
server_key: "96dabb8fe8d7713e3cd41cf83ca3e656"
server_key_iv: "2851c61c08f755c280767634e3f1b983"
server_key_version: "encryptionkey@0"
I20260927 02:44:44.858304 13998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:44.867805 13998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:44.869916 13998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:44.870052 13998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:44.873814 13998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42629
I20260927 02:44:44.873848 14065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42629 every 8 connection(s)
I20260927 02:44:44.874158 13998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:44.874409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 13998
I20260927 02:44:44.874526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:44.874693 392 external_mini_cluster.cc:1428] Setting key bcf091a5c2fd5b1416fe36d21689cc7c
I20260927 02:44:44.875032 14066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:44.878108 14066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc: Bootstrap starting.
I20260927 02:44:44.878703 14066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:44.878944 14066 log.cc:824] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:44.879649 14066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc: No bootstrap required, opened a new log
Sep 27 02:44:44 dist-test-slave-44kr krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:44.881632 14066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62478bbefd094758bd3128340e9adddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } }
I20260927 02:44:44.881817 14066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:44.881865 14066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62478bbefd094758bd3128340e9adddc, State: Initialized, Role: FOLLOWER
I20260927 02:44:44.881999 14066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62478bbefd094758bd3128340e9adddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } }
I20260927 02:44:44.882079 14066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:44.882162 14066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:44.882218 14066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:44.882853 14066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62478bbefd094758bd3128340e9adddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } }
I20260927 02:44:44.882948 14066 leader_election.cc:302] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62478bbefd094758bd3128340e9adddc; no voters:
I20260927 02:44:44.883116 14066 leader_election.cc:288] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:44.883174 14071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:44.883317 14071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [term 1 LEADER]: Becoming Leader. State: Replica: 62478bbefd094758bd3128340e9adddc, State: Running, Role: LEADER
I20260927 02:44:44.883378 14066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:44.883482 14071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62478bbefd094758bd3128340e9adddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } }
I20260927 02:44:44.883945 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62478bbefd094758bd3128340e9adddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62478bbefd094758bd3128340e9adddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } } }
I20260927 02:44:44.884025 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:44.884299 14080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:44.884645 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62478bbefd094758bd3128340e9adddc. Latest consensus state: current_term: 1 leader_uuid: "62478bbefd094758bd3128340e9adddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62478bbefd094758bd3128340e9adddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } } }
I20260927 02:44:44.884729 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:44.884897 14080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:44.885811 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:44.875202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.0.98.62:42629)
0927 02:44:44.875379 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:44.875389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:44.875422 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:44:44.876076 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:44.876085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:44.876151 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:44.876313 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:44.876890 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.878228 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.879145 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:44.879637 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:44.879757 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:44.882174 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:44.882178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:44.882179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:44.882191 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:44.884066 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:44.884497 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:44.884500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:44.884500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:44.884604 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:44.884806 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:44.884807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:44.884808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:44.885117 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:44.885249 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:44.885489 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 02:44:44.885625 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260927 02:44:44.886659 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:44:44.888830 14080 catalog_manager.cc:1347] Generated new cluster ID: 99ded4088691479eb61e57964be151b6
I20260927 02:44:44.888881 14080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:44.904172 14080 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:44.904821 14080 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:44.908836 14080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62478bbefd094758bd3128340e9adddc: Generated new TSK 0
I20260927 02:44:44.909017 14080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:44:44.940361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 13998
2026-09-27T02:44:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1252 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:44:44 dist-test-slave-44kr krb5kdc[14093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:44:44 dist-test-slave-44kr krb5kdc[14093](info): set up 2 sockets
Sep 27 02:44:44 dist-test-slave-44kr krb5kdc[14093](info): commencing operation
krb5kdc: starting...
W20260927 02:44:45.900218 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:44:45 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:44:46.016783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46627
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:46.096103 14106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:46.096311 14106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:46.096331 14106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:46.097468 14106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:46.097512 14106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:46.097524 14106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:46.097535 14106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:46.097546 14106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:46.098963 14106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:46.099162 14106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:46.099331 14106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:46.104089 14112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:46.104097 14114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:46.104095 14111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:46.104182 14106 server_base.cc:1034] running on GCE node
I20260927 02:44:46.104638 14106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:46.104933 14106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:46.106076 14106 hybrid_clock.cc:648] HybridClock initialized: now 1790477086106061 us; error 28 us; skew 500 ppm
Sep 27 02:44:46 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:46.111303 14106 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:46.111766 14106 webserver.cc:460] Webserver started at http://127.0.98.62:34575/ using document root <none> and password file <none>
I20260927 02:44:46.111965 14106 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:46.112010 14106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:46.112116 14106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:46.113235 14106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "4d305b94633148258630996f49fa8328"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "659dee896b1ae459497ac9ace5220afc"
server_key_iv: "fd73ec01783a9586b323b219be767f03"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.113564 14106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d305b94633148258630996f49fa8328"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "659dee896b1ae459497ac9ace5220afc"
server_key_iv: "fd73ec01783a9586b323b219be767f03"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.114861 14106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:44:46.115670 14121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.115875 14106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:46.115964 14106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "4d305b94633148258630996f49fa8328"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "659dee896b1ae459497ac9ace5220afc"
server_key_iv: "fd73ec01783a9586b323b219be767f03"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.116024 14106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:46.150864 14106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:46.153147 14106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:46.153268 14106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:46.156888 14106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46627
I20260927 02:44:46.156951 14173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46627 every 8 connection(s)
I20260927 02:44:46.157272 14106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:46.158316 14174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:46.160548 14174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Bootstrap starting.
I20260927 02:44:46.161077 14174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:46.161332 14174 log.cc:824] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:46.161937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 14106
I20260927 02:44:46.162040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:44:46.162034 14174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: No bootstrap required, opened a new log
I20260927 02:44:46.162225 392 external_mini_cluster.cc:1428] Setting key 4fb7c4a34130ce736350e386cf0820d6
I20260927 02:44:46.164310 14174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:46.164494 14174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:46.164532 14174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d305b94633148258630996f49fa8328, State: Initialized, Role: FOLLOWER
I20260927 02:44:46.164650 14174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:46.164708 14174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:46.164737 14174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:46.164783 14174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:46.165299 14174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:46.165398 14174 leader_election.cc:302] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d305b94633148258630996f49fa8328; no voters:
I20260927 02:44:46.165557 14174 leader_election.cc:288] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:44:46.165591 14179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:46.165742 14179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 1 LEADER]: Becoming Leader. State: Replica: 4d305b94633148258630996f49fa8328, State: Running, Role: LEADER
I20260927 02:44:46.165818 14174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:46.165870 14179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:46.166241 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d305b94633148258630996f49fa8328. Latest consensus state: current_term: 1 leader_uuid: "4d305b94633148258630996f49fa8328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } } }
I20260927 02:44:46.166464 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:46.166400 14181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d305b94633148258630996f49fa8328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } } }
I20260927 02:44:46.166857 14181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:46.167013 14188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:46.167536 14188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:44:46 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:46.169675 14188 catalog_manager.cc:1347] Generated new cluster ID: 713d31fd03c341f6a6505193e185d5ed
I20260927 02:44:46.169718 14188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:46.173475 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.162698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.0.98.62:46627)
0927 02:44:46.162852 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.162860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.162885 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:44:46.163327 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.163334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.163381 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.163491 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:46.164282 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.165557 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.166585 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.167210 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.167339 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.169808 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:46.169812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:46.169813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:46.169826 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:46.171789 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.172194 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.172197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.172198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.172308 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.172519 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.172520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.172521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.172855 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:46.173012 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.173110 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.173251 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260927 02:44:46.174317 14188 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:44:46.174803 14188 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:46.184545 14188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Generated new TSK 0
I20260927 02:44:46.184784 14188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:44:46.211879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46627
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:46.295404 14200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:46.295619 14200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:46.295642 14200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:46.296859 14200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:46.296901 14200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:46.296927 14200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:46.298377 14200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:46.298599 14200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:46.298813 14200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:46.303601 14206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:46.303586 14205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:46.303644 14200 server_base.cc:1034] running on GCE node
W20260927 02:44:46.303586 14208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:46.303947 14200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:46.304147 14200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:46.305294 14200 hybrid_clock.cc:648] HybridClock initialized: now 1790477086305282 us; error 29 us; skew 500 ppm
Sep 27 02:44:46 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:46.307578 14200 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:46.308005 14200 webserver.cc:460] Webserver started at http://127.0.98.1:45821/ using document root <none> and password file <none>
I20260927 02:44:46.308195 14200 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:46.308239 14200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:46.308341 14200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:46.309379 14200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "01e160c6587c4537a375afe58f592510"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "cd0055cee0f92559dd96a2335d0f7595"
server_key_iv: "7f6787f9797eafff28701d3ea8d8ef10"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.309695 14200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01e160c6587c4537a375afe58f592510"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "cd0055cee0f92559dd96a2335d0f7595"
server_key_iv: "7f6787f9797eafff28701d3ea8d8ef10"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.310858 14200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:44:46.311457 14215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.311584 14200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:44:46.311661 14200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "01e160c6587c4537a375afe58f592510"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "cd0055cee0f92559dd96a2335d0f7595"
server_key_iv: "7f6787f9797eafff28701d3ea8d8ef10"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.311731 14200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:46.333123 14200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:46.334882 14200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:46.335016 14200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:46.335284 14200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:46.335584 14200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:46.335618 14200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.335646 14200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:46.335667 14200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.341569 14200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44033
I20260927 02:44:46.341642 14328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44033 every 8 connection(s)
I20260927 02:44:46.341936 14200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:46.346961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 14200
I20260927 02:44:46.347118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:44:46.347319 392 external_mini_cluster.cc:1428] Setting key e72a7fe4cad30f73f7bc881977255fbf
Sep 27 02:44:46 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:46.351637 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.342623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44907 (local address 127.0.98.62:46627)
0927 02:44:46.342664 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.342666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.342858 (+ 192us) server_negotiation.cc:407] Connection header received
0927 02:44:46.343348 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.343349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.343361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.343412 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:46.344349 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.344874 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.345911 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.346125 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.346188 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.348720 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:46.348721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:46.348722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:46.348729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:46.350228 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.350614 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.350615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.350615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.350702 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.350916 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.350916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.350917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.351044 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:46.351107 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.351460 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.351544 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:44:46.351738 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.342599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:44907)
0927 02:44:46.342743 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.342782 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.343253 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.343470 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.343473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.344036 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:46.344235 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.344240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.344947 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.344947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.345806 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.345809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.346195 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.346196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.346337 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.346798 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:46.346807 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:46.348624 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:46.350316 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:46.350322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:46.350324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:46.350538 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:46.350762 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:46.350762 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:46.350763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:46.350864 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:46.351140 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:46.351146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:46.351308 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 02:44:46.351421 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 02:44:46.351564 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260927 02:44:46.352188 14329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46627
I20260927 02:44:46.352300 14329 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:46.352530 14329 heartbeater.cc:511] Master 127.0.98.62:46627 requested a full tablet report, sending...
I20260927 02:44:46.353025 14138 ts_manager.cc:194] Registered new tserver with Master: 01e160c6587c4537a375afe58f592510 (127.0.98.1:44033)
I20260927 02:44:46.353600 14138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44907
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:44:46.371762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46627
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:46.453728 14334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:46.453933 14334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:46.453953 14334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:46.455077 14334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:46.455121 14334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:46.455152 14334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:46.456624 14334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:46.456836 14334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:46.457056 14334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:46.463058 14340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:46.463097 14339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:46.463052 14342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:46.463349 14334 server_base.cc:1034] running on GCE node
I20260927 02:44:46.463546 14334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:46.463783 14334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:46.464927 14334 hybrid_clock.cc:648] HybridClock initialized: now 1790477086464914 us; error 25 us; skew 500 ppm
Sep 27 02:44:46 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:46.467583 14334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:46.468060 14334 webserver.cc:460] Webserver started at http://127.0.98.2:44907/ using document root <none> and password file <none>
I20260927 02:44:46.468343 14334 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:46.468431 14334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:46.468559 14334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:46.469714 14334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "f3f230c42067402aa64aebb864f08fae"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "6de0ffed9bb5e9440b98358b7e575175"
server_key_iv: "94649a62cfbdf8b76fdded52ddfd87ba"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.470052 14334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f3f230c42067402aa64aebb864f08fae"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "6de0ffed9bb5e9440b98358b7e575175"
server_key_iv: "94649a62cfbdf8b76fdded52ddfd87ba"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.471315 14334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:44:46.472069 14349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.472230 14334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:44:46.472317 14334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "f3f230c42067402aa64aebb864f08fae"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "6de0ffed9bb5e9440b98358b7e575175"
server_key_iv: "94649a62cfbdf8b76fdded52ddfd87ba"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.472380 14334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:46.482127 14334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:46.484082 14334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:46.484205 14334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:46.484439 14334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:46.484759 14334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:46.484797 14334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.484818 14334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:46.484830 14334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.490278 14334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39485
I20260927 02:44:46.490346 14462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39485 every 8 connection(s)
I20260927 02:44:46.490631 14334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:44:46 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:46.496695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 14334
I20260927 02:44:46.496799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:44:46.496994 392 external_mini_cluster.cc:1428] Setting key 47cad5c7b19fc36e21b21fa1547d7b5f
I20260927 02:44:46.499840 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.491398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43201 (local address 127.0.98.62:46627)
0927 02:44:46.491442 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.491444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.491668 (+ 224us) server_negotiation.cc:407] Connection header received
0927 02:44:46.492173 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.492175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.492186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.492237 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:46.493228 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.493709 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.494677 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.494880 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.494937 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.497137 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:46.497139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:46.497139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:46.497147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:46.498527 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.498990 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.498991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.498991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.499046 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.499263 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.499264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.499264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.499369 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:46.499425 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.499673 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.499751 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:44:46.500042 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.491366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:43201)
0927 02:44:46.491548 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.491584 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.492075 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.492318 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.492322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.492940 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:46.493133 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.493138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.493791 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.493792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.494598 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.494600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.494950 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.494951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.495074 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.495536 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:46.495545 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:46.497038 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:46.498607 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:46.498613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:46.498615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:46.498917 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:46.499101 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:46.499101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:46.499102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:46.499205 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:46.499421 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:46.499426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:46.499560 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 02:44:46.499646 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:44:46.499811 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260927 02:44:46.500581 14463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46627
I20260927 02:44:46.500682 14463 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:46.500877 14463 heartbeater.cc:511] Master 127.0.98.62:46627 requested a full tablet report, sending...
I20260927 02:44:46.501271 14138 ts_manager.cc:194] Registered new tserver with Master: f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485)
I20260927 02:44:46.501770 14138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43201
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:44:46.524319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46627
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:46.603104 14468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:46.603307 14468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:46.603328 14468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:46.604475 14468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:46.604516 14468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:46.604540 14468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:46.605963 14468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:46.606204 14468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:46.606426 14468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:46.611285 14476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:46.611299 14474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:46.611372 14473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:46.611536 14468 server_base.cc:1034] running on GCE node
I20260927 02:44:46.611727 14468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:46.611949 14468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:46.613113 14468 hybrid_clock.cc:648] HybridClock initialized: now 1790477086613077 us; error 47 us; skew 500 ppm
Sep 27 02:44:46 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:46.615700 14468 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:46.616119 14468 webserver.cc:460] Webserver started at http://127.0.98.3:36063/ using document root <none> and password file <none>
I20260927 02:44:46.616377 14468 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:46.616463 14468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:46.616587 14468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:44:46.617609 14468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ee5870f4e7541108d6b4075b7c84390"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "99d05f7310c76ada59af8aa43a9cac28"
server_key_iv: "4ae2602452359721e25d457a38070f67"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.617929 14468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ee5870f4e7541108d6b4075b7c84390"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "99d05f7310c76ada59af8aa43a9cac28"
server_key_iv: "4ae2602452359721e25d457a38070f67"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.619166 14468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:44:46.619920 14483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.620154 14468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:46.620249 14468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "5ee5870f4e7541108d6b4075b7c84390"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "99d05f7310c76ada59af8aa43a9cac28"
server_key_iv: "4ae2602452359721e25d457a38070f67"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.620311 14468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:46.639904 14468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:46.642618 14468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:46.642772 14468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:46.643013 14468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:46.643362 14468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:46.643402 14468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.643432 14468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:46.643455 14468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.649019 14468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44041
I20260927 02:44:46.649091 14596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44041 every 8 connection(s)
I20260927 02:44:46.649371 14468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:44:46 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:46.658769 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.650143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44211 (local address 127.0.98.62:46627)
0927 02:44:46.650222 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.650225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.650394 (+ 169us) server_negotiation.cc:407] Connection header received
0927 02:44:46.650893 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.650894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.650906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.650965 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:46.651952 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.652512 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.653509 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.653714 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.653772 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.655913 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:46.655915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:46.655915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:46.655923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:46.657451 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.657862 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.657863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.657863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.657917 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.658134 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.658135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.658135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.658263 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:46.658322 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.658582 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.658677 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:44:46.658958 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.650077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:44211)
0927 02:44:46.650269 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.650306 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.650799 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.651036 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.651040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.651651 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:46.651858 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.651863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.652601 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.652602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.653431 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.653434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.653783 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.653784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.653904 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.654409 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:46.654417 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:46.655822 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:46.657523 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:46.657531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:46.657533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:46.657793 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:46.657971 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:46.657972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:46.657972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:46.658079 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:46.658322 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:46.658329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:46.658465 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 02:44:46.658555 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:44:46.658728 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260927 02:44:46.659138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 14468
I20260927 02:44:46.659241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:44:46.659438 392 external_mini_cluster.cc:1428] Setting key b3fa75593aed40f07385a08e10b68602
I20260927 02:44:46.659554 14597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46627
I20260927 02:44:46.659655 14597 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:46.659909 14597 heartbeater.cc:511] Master 127.0.98.62:46627 requested a full tablet report, sending...
I20260927 02:44:46.660296 14138 ts_manager.cc:194] Registered new tserver with Master: 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041)
I20260927 02:44:46.660683 14138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44211
I20260927 02:44:46.671161 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:44:46.678432 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.672612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.0.98.62:46627)
0927 02:44:46.672683 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.672686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.672776 (+ 90us) server_negotiation.cc:407] Connection header received
0927 02:44:46.672836 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.672837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.672848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.672896 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:46.673460 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.673998 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.674717 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.674924 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.674979 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.676148 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:46.676149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:46.676149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:46.676157 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:46.677419 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.677714 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.677716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.677716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.677767 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:46.677926 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:46.677927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:46.677927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:46.678072 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:46.678157 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.678265 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.678341 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:44:46.679857 14138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:46.681529 14138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:46.687556 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.683586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39485 (local address 127.0.0.1:40630)
0927 02:44:46.684159 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.684168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.684232 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.684475 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.684476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.684478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:46.684522 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.684524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.685383 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.685384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.686414 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.686416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.687245 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.687246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.687325 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.687340 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 02:44:46.687417 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:44:46.687473 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":5,"thread_start_us":53,"threads_started":1}
I20260927 02:44:46.687559 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.684017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.0.98.2:39485)
0927 02:44:46.684177 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.684182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.684230 (+ 48us) server_negotiation.cc:407] Connection header received
0927 02:44:46.684293 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.684295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.684309 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.684379 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:46.684577 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.685293 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.686485 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.687158 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.687237 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.687304 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.687435 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.687472 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260927 02:44:46.687830 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.683786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.0.98.1:44033)
0927 02:44:46.683998 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.684003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.684014 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:44:46.684047 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.684048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.684061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.684142 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:46.684471 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.685578 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.686810 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.687426 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.687483 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.687535 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.687715 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.687755 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":112,"threads_started":1}
I20260927 02:44:46.688023 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.683822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.0.98.3:44041)
0927 02:44:46.684271 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.684274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.684427 (+ 153us) server_negotiation.cc:407] Connection header received
0927 02:44:46.684538 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.684541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.684555 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.684618 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:46.684714 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.685843 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.686957 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.687633 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.687707 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.687758 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.687911 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.687950 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":82,"threads_started":1}
I20260927 02:44:46.687992 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.683591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44041 (local address 127.0.0.1:40434)
0927 02:44:46.684344 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.684352 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.684454 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.684623 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.684624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.684625 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:46.684661 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.684663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.685937 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.685938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.686879 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.686882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.687789 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.687790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.687836 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.687840 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:46.687885 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:44:46.687925 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260927 02:44:46.688078 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.683586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44033 (local address 127.0.0.1:49038)
0927 02:44:46.683832 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.683877 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.683942 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.684156 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.684159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.684161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:46.684351 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.684354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.685713 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.685714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.686727 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.686730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.687566 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.687566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.687621 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.687626 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:44:46.687929 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 02:44:46.687980 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":33,"thread_start_us":56,"threads_started":1}
I20260927 02:44:46.688797 14531 tablet_service.cc:1467] Processing CreateTablet for tablet b2706b6f91b946328928937a9d1c5090 (DEFAULT_TABLE table=test-table [id=b466f46a98d549a4877e3bd7eebff926]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:46.688827 14397 tablet_service.cc:1467] Processing CreateTablet for tablet b2706b6f91b946328928937a9d1c5090 (DEFAULT_TABLE table=test-table [id=b466f46a98d549a4877e3bd7eebff926]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:46.688867 14263 tablet_service.cc:1467] Processing CreateTablet for tablet b2706b6f91b946328928937a9d1c5090 (DEFAULT_TABLE table=test-table [id=b466f46a98d549a4877e3bd7eebff926]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:46.689128 14531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2706b6f91b946328928937a9d1c5090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:46.689162 14397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2706b6f91b946328928937a9d1c5090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:46.689162 14263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2706b6f91b946328928937a9d1c5090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:46.691715 14611 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: Bootstrap starting.
I20260927 02:44:46.692252 14612 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Bootstrap starting.
I20260927 02:44:46.692320 14611 tablet_bootstrap.cc:654] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:46.692942 14611 log.cc:824] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:46.692976 14612 tablet_bootstrap.cc:654] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:46.693464 14612 log.cc:824] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:46.693744 14611 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: No bootstrap required, opened a new log
I20260927 02:44:46.693797 14613 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Bootstrap starting.
I20260927 02:44:46.693814 14611 ts_tablet_manager.cc:1397] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:46.694116 14612 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: No bootstrap required, opened a new log
I20260927 02:44:46.694180 14612 ts_tablet_manager.cc:1397] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:44:46.694422 14613 tablet_bootstrap.cc:654] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:46.695092 14613 log.cc:824] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:46.695904 14613 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: No bootstrap required, opened a new log
I20260927 02:44:46.695780 14611 raft_consensus.cc:348] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:46.695950 14611 raft_consensus.cc:374] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:46.695976 14613 ts_tablet_manager.cc:1397] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:44:46.695983 14611 raft_consensus.cc:729] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee5870f4e7541108d6b4075b7c84390, State: Initialized, Role: FOLLOWER
I20260927 02:44:46.695933 14612 raft_consensus.cc:348] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:46.696110 14612 raft_consensus.cc:374] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:46.696147 14612 raft_consensus.cc:729] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e160c6587c4537a375afe58f592510, State: Initialized, Role: FOLLOWER
I20260927 02:44:46.696142 14611 consensus_queue.cc:260] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:46.696295 14612 consensus_queue.cc:260] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:46.696449 14611 ts_tablet_manager.cc:1428] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:46.696508 14597 heartbeater.cc:503] Master 127.0.98.62:46627 was elected leader, sending a full tablet report...
I20260927 02:44:46.696591 14612 ts_tablet_manager.cc:1428] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:44:46.696738 14329 heartbeater.cc:503] Master 127.0.98.62:46627 was elected leader, sending a full tablet report...
I20260927 02:44:46.698210 14613 raft_consensus.cc:348] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:46.698345 14613 raft_consensus.cc:374] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:46.698366 14613 raft_consensus.cc:729] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f230c42067402aa64aebb864f08fae, State: Initialized, Role: FOLLOWER
I20260927 02:44:46.698446 14613 consensus_queue.cc:260] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:46.698642 14613 ts_tablet_manager.cc:1428] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:44:46.698707 14463 heartbeater.cc:503] Master 127.0.98.62:46627 was elected leader, sending a full tablet report...
I20260927 02:44:46.735747 14617 raft_consensus.cc:482] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:46.735893 14617 raft_consensus.cc:504] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:46.736282 14617 leader_election.cc:288] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01e160c6587c4537a375afe58f592510 (127.0.98.1:44033), f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485)
I20260927 02:44:46.739099 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.736571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36677 (local address 127.0.98.2:39485)
0927 02:44:46.736617 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.736618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.736765 (+ 147us) server_negotiation.cc:407] Connection header received
0927 02:44:46.736811 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.736812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.736825 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.736880 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:46.737012 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.737459 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.738425 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.738793 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.738843 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.738853 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.738992 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.739029 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:44:46.739130 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.736577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34017 (local address 127.0.98.1:44033)
0927 02:44:46.736617 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.736619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.736629 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:44:46.736661 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.736662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.736673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.736749 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:46.736902 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.737450 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.738433 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.738806 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.738874 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.738884 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.739005 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.739043 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:44:46.739132 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.736493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44033 (local address 127.0.98.3:34017)
0927 02:44:46.736546 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.736557 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.736599 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.736762 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.736763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.736765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:46.736837 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.736839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.737526 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.737527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.738343 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.738345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.738882 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.738883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.738942 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.738945 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:44:46.738992 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:44:46.739029 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":48}
I20260927 02:44:46.739140 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.736493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39485 (local address 127.0.98.3:36677)
0927 02:44:46.736692 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.736702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:46.736758 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:46.736881 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:46.736882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:46.736883 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:46.736954 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.736955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.737532 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.737533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.738342 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:46.738345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.738862 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.738863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:46.738928 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.738932 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:46.738979 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:44:46.739017 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":12,"thread_start_us":127,"threads_started":1}
I20260927 02:44:46.739449 14283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2706b6f91b946328928937a9d1c5090" candidate_uuid: "5ee5870f4e7541108d6b4075b7c84390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e160c6587c4537a375afe58f592510" is_pre_election: true
I20260927 02:44:46.739619 14283 raft_consensus.cc:2454] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee5870f4e7541108d6b4075b7c84390 in term 0.
I20260927 02:44:46.739603 14417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2706b6f91b946328928937a9d1c5090" candidate_uuid: "5ee5870f4e7541108d6b4075b7c84390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f230c42067402aa64aebb864f08fae" is_pre_election: true
I20260927 02:44:46.739722 14417 raft_consensus.cc:2454] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee5870f4e7541108d6b4075b7c84390 in term 0.
I20260927 02:44:46.739845 14485 leader_election.cc:302] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e160c6587c4537a375afe58f592510, 5ee5870f4e7541108d6b4075b7c84390; no voters:
I20260927 02:44:46.739974 14617 raft_consensus.cc:2781] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:46.740034 14617 raft_consensus.cc:482] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:46.740065 14617 raft_consensus.cc:3037] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:46.740801 14617 raft_consensus.cc:504] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:46.740907 14617 leader_election.cc:288] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [CANDIDATE]: Term 1 election: Requested vote from peers 01e160c6587c4537a375afe58f592510 (127.0.98.1:44033), f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485)
I20260927 02:44:46.741124 14283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2706b6f91b946328928937a9d1c5090" candidate_uuid: "5ee5870f4e7541108d6b4075b7c84390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e160c6587c4537a375afe58f592510"
W20260927 02:44:46.741158 14464 tablet.cc:2369] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:44:46.741147 14417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2706b6f91b946328928937a9d1c5090" candidate_uuid: "5ee5870f4e7541108d6b4075b7c84390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f230c42067402aa64aebb864f08fae"
I20260927 02:44:46.741201 14283 raft_consensus.cc:3037] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:46.741238 14417 raft_consensus.cc:3037] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:46.741916 14283 raft_consensus.cc:2454] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee5870f4e7541108d6b4075b7c84390 in term 1.
I20260927 02:44:46.741981 14417 raft_consensus.cc:2454] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee5870f4e7541108d6b4075b7c84390 in term 1.
I20260927 02:44:46.742080 14485 leader_election.cc:302] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e160c6587c4537a375afe58f592510, 5ee5870f4e7541108d6b4075b7c84390; no voters:
I20260927 02:44:46.742172 14617 raft_consensus.cc:2781] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:46.742319 14617 raft_consensus.cc:686] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee5870f4e7541108d6b4075b7c84390, State: Running, Role: LEADER
I20260927 02:44:46.742393 14617 consensus_queue.cc:237] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:46.743188 14136 catalog_manager.cc:5495] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5ee5870f4e7541108d6b4075b7c84390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } health_report { overall_health: UNKNOWN } } }
I20260927 02:44:46.761318 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.758920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.0.98.3:44041)
0927 02:44:46.758977 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:46.758979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:46.758987 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:44:46.759025 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:46.759025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:46.759045 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:46.759171 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:46.759363 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.759842 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.760628 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:46.760812 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:46.760855 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:46.761021 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:46.761131 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:46.761193 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:46.761230 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 02:44:46.761249 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:44:46.763130 14283 raft_consensus.cc:1261] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Refusing update from remote peer 5ee5870f4e7541108d6b4075b7c84390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:46.763155 14417 raft_consensus.cc:1261] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 1 FOLLOWER]: Refusing update from remote peer 5ee5870f4e7541108d6b4075b7c84390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:46.763480 14621 consensus_queue.cc:1035] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:46.763644 14617 consensus_queue.cc:1035] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:46.766299 14624 mvcc.cc:205] Tried to move back new op lower bound from 7333794147379036160 to 7333794147297390592. Current Snapshot: MvccSnapshot[applied={T|T < 7333794147379036160}]
I20260927 02:44:46.767037 14625 mvcc.cc:205] Tried to move back new op lower bound from 7333794147379036160 to 7333794147297390592. Current Snapshot: MvccSnapshot[applied={T|T < 7333794147379036160 or (T in {7333794147379036160})}]
I20260927 02:44:46.769304 14136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:46.769382 14136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:46.770124 14136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Sending DeleteTablet for 3 replicas of tablet b2706b6f91b946328928937a9d1c5090
I20260927 02:44:46.770498 14263 tablet_service.cc:1514] Processing DeleteTablet for tablet b2706b6f91b946328928937a9d1c5090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49038
I20260927 02:44:46.770529 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet b2706b6f91b946328928937a9d1c5090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260927 02:44:46.770694 14531 tablet_service.cc:1514] Processing DeleteTablet for tablet b2706b6f91b946328928937a9d1c5090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40434
I20260927 02:44:46.770752 14633 tablet_replica.cc:330] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: stopping tablet replica
I20260927 02:44:46.770844 14633 raft_consensus.cc:2229] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:46.770947 14633 raft_consensus.cc:2258] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:46.770954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 14200
W20260927 02:44:46.775624 14123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44033 (error 108)
I20260927 02:44:46.775709 14635 tablet_replica.cc:330] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: stopping tablet replica
I20260927 02:44:46.775807 14635 raft_consensus.cc:2229] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:46.775950 14635 raft_consensus.cc:2258] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:46.776161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 14334
I20260927 02:44:46.776317 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.776114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44033 (local address 127.0.0.1:49042)
0927 02:44:46.776245 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.776286 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260927 02:44:46.777143 14123 catalog_manager.cc:4584] TS 01e160c6587c4537a375afe58f592510 (127.0.98.1:44033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2706b6f91b946328928937a9d1c5090: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44033: connect: Connection refused (error 111)
I20260927 02:44:46.778373 14635 ts_tablet_manager.cc:1905] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:46.779906 14635 ts_tablet_manager.cc:1918] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:46.779987 14635 log.cc:1196] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/b2706b6f91b946328928937a9d1c5090
I20260927 02:44:46.780257 14635 ts_tablet_manager.cc:1939] T b2706b6f91b946328928937a9d1c5090 P 5ee5870f4e7541108d6b4075b7c84390: Deleting consensus metadata
I20260927 02:44:46.780707 14122 catalog_manager.cc:4841] TS 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041): tablet b2706b6f91b946328928937a9d1c5090 (table test-table [id=b466f46a98d549a4877e3bd7eebff926]) successfully deleted
I20260927 02:44:46.782335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 14468
I20260927 02:44:46.782370 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:46.782260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39485 (local address 127.0.0.1:40638)
0927 02:44:46.782307 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:46.782339 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:44:46.786641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 14106
I20260927 02:44:46.791131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46627
--webserver_interface=127.0.98.62
--webserver_port=34575
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:46.869184 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:46.869405 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:46.869434 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:46.870694 14637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:46.870754 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:46.870771 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:46.870787 14637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:46.870801 14637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:46.872443 14637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:46.872706 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:46.872933 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:46.881385 14643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:46.881487 14637 server_base.cc:1034] running on GCE node
W20260927 02:44:46.881395 14644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:46.881383 14646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:46.881855 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:46.882169 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:46.883322 14637 hybrid_clock.cc:648] HybridClock initialized: now 1790477086883308 us; error 29 us; skew 500 ppm
Sep 27 02:44:46 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:46.885829 14637 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:46.886245 14637 webserver.cc:460] Webserver started at http://127.0.98.62:34575/ using document root <none> and password file <none>
I20260927 02:44:46.886456 14637 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:46.886511 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:46.887861 14637 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:46.888496 14653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.888648 14637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:46.888739 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "4d305b94633148258630996f49fa8328"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "659dee896b1ae459497ac9ace5220afc"
server_key_iv: "fd73ec01783a9586b323b219be767f03"
server_key_version: "encryptionkey@0"
I20260927 02:44:46.889117 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:46.902041 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:46.904142 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:46.904278 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:46.907945 14637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46627
I20260927 02:44:46.907986 14705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46627 every 8 connection(s)
I20260927 02:44:46.908341 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:46.909940 14706 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:44:46.910548 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Bootstrap starting.
I20260927 02:44:46.915748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 14637
I20260927 02:44:46.916002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44033
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46627
--webserver_port=45821
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:46.918426 14706 log.cc:824] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:46.920228 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:44:46.920509 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Bootstrap complete.
I20260927 02:44:46.922605 14706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:46.922842 14706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d305b94633148258630996f49fa8328, State: Initialized, Role: FOLLOWER
I20260927 02:44:46.922983 14706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:46.923044 14706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:46.923089 14706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:46.923135 14706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:44:46.924151 14706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:46.924238 14706 leader_election.cc:302] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d305b94633148258630996f49fa8328; no voters:
I20260927 02:44:46.924420 14706 leader_election.cc:288] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:44:46.924455 14710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:44:46.924614 14710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 2 LEADER]: Becoming Leader. State: Replica: 4d305b94633148258630996f49fa8328, State: Running, Role: LEADER
I20260927 02:44:46.924693 14706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:46.924805 14710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:46.925093 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d305b94633148258630996f49fa8328. Latest consensus state: current_term: 2 leader_uuid: "4d305b94633148258630996f49fa8328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } } }
I20260927 02:44:46.925161 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:46.925240 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4d305b94633148258630996f49fa8328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } } }
I20260927 02:44:46.925287 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:46.925570 14715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:46.926422 14715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:46.926525 14715 catalog_manager.cc:1259] Loaded cluster ID: 713d31fd03c341f6a6505193e185d5ed
I20260927 02:44:46.926761 14715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:44:46.927050 14715 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 02:44:46.932260 14715 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 02:44:46.934675 14715 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:44:46.934882 14715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Loaded TSK: 0
I20260927 02:44:46.935096 14715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:44:46.998481 14708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:46.998679 14708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:46.998701 14708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:46.999850 14708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:46.999891 14708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:46.999917 14708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:47.001363 14708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45821
--tserver_master_addrs=127.0.98.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:47.001590 14708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:47.001811 14708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:47.007684 14733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.007696 14736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.007697 14734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:47.007911 14708 server_base.cc:1034] running on GCE node
I20260927 02:44:47.008211 14708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:47.008496 14708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:47.009675 14708 hybrid_clock.cc:648] HybridClock initialized: now 1790477087009651 us; error 39 us; skew 500 ppm
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:47.012079 14708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:47.012573 14708 webserver.cc:460] Webserver started at http://127.0.98.1:45821/ using document root <none> and password file <none>
I20260927 02:44:47.012809 14708 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:47.012862 14708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:47.014168 14708 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:47.014820 14743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.014962 14708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:44:47.015041 14708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "01e160c6587c4537a375afe58f592510"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "cd0055cee0f92559dd96a2335d0f7595"
server_key_iv: "7f6787f9797eafff28701d3ea8d8ef10"
server_key_version: "encryptionkey@0"
I20260927 02:44:47.015384 14708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:47.032928 14708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:47.035056 14708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:47.035197 14708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:47.035409 14708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:47.035882 14750 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:44:47.037320 14708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:44:47.037369 14708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:44:47.037395 14708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:44:47.038000 14708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:44:47.038040 14708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:47.038110 14750 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Bootstrap starting.
I20260927 02:44:47.045192 14708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44033
I20260927 02:44:47.045368 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44033 every 8 connection(s)
I20260927 02:44:47.045750 14708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:47.048558 14750 log.cc:824] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:47.050540 14750 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:44:47.050793 14750 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Bootstrap complete.
I20260927 02:44:47.050873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 14708
I20260927 02:44:47.051144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39485
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46627
--webserver_port=44907
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:47.051630 14750 ts_tablet_manager.cc:1397] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260927 02:44:47.052928 14750 raft_consensus.cc:348] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:47.053241 14750 raft_consensus.cc:729] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e160c6587c4537a375afe58f592510, State: Initialized, Role: FOLLOWER
I20260927 02:44:47.053407 14750 consensus_queue.cc:260] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.053655 14750 ts_tablet_manager.cc:1428] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
I20260927 02:44:47.056809 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.046536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49341 (local address 127.0.98.62:46627)
0927 02:44:47.046729 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.046739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.046980 (+ 241us) server_negotiation.cc:407] Connection header received
0927 02:44:47.047693 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.047702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.047785 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.047964 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:47.048802 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.049632 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.050661 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.051254 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.051363 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.053727 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:47.053731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:47.053732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:47.053744 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:47.055227 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.055587 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.055590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.055590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.055677 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.055911 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.055912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.055912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.056119 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:47.056188 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.056485 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.056614 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260927 02:44:47.056809 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.046438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:49341)
0927 02:44:47.046815 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.046856 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.047436 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.048010 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.048014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.048420 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:47.048694 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.048702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.049755 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.049756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.050568 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.050570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.051473 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.051474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.051603 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.052176 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:47.052193 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:47.053637 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:47.055299 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.055305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.055307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.055513 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.055744 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.055745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.055745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.055853 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.056190 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:47.056196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:47.056320 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 02:44:47.056437 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:44:47.056614 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":81,"threads_started":1}
I20260927 02:44:47.057386 14858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46627
I20260927 02:44:47.057466 14858 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:47.057623 14858 heartbeater.cc:511] Master 127.0.98.62:46627 requested a full tablet report, sending...
I20260927 02:44:47.058260 14670 ts_manager.cc:194] Registered new tserver with Master: 01e160c6587c4537a375afe58f592510 (127.0.98.1:44033)
I20260927 02:44:47.059331 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49341
I20260927 02:44:47.060088 14858 heartbeater.cc:503] Master 127.0.98.62:46627 was elected leader, sending a full tablet report...
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:44:47.067422 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.058784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44033 (local address 127.0.0.1:49046)
0927 02:44:47.058974 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.059002 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.059066 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.059249 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.059252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.059544 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:47.059669 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.059672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.061408 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.061409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.061956 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.061957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.062549 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.062553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.062617 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.063054 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:47.063058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:47.064242 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:47.066138 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.066145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.066147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.066335 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.066584 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.066585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.066585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.066650 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.067002 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:47.067008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:47.067065 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 02:44:47.067236 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 02:44:47.067341 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260927 02:44:47.067477 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.058858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.0.98.1:44033)
0927 02:44:47.059014 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.059018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.059051 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:44:47.059144 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.059146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.059162 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.059243 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:47.060236 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.061335 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.062021 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.062459 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.062524 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.064318 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:47.064324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:47.064325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:47.064332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:47.066052 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.066404 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.066407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.066407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.066468 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.066719 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.066720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.066720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.066908 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:47.066999 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.067283 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.067379 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260927 02:44:47.067860 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet b2706b6f91b946328928937a9d1c5090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49046
I20260927 02:44:47.068478 14867 tablet_replica.cc:330] stopping tablet replica
I20260927 02:44:47.068578 14867 raft_consensus.cc:2229] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:47.068642 14867 raft_consensus.cc:2258] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:47.068920 14867 ts_tablet_manager.cc:1905] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:47.070886 14867 ts_tablet_manager.cc:1918] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:47.071007 14867 log.cc:1196] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/b2706b6f91b946328928937a9d1c5090
I20260927 02:44:47.071220 14867 ts_tablet_manager.cc:1939] T b2706b6f91b946328928937a9d1c5090 P 01e160c6587c4537a375afe58f592510: Deleting consensus metadata
I20260927 02:44:47.071700 14655 catalog_manager.cc:4841] TS 01e160c6587c4537a375afe58f592510 (127.0.98.1:44033): tablet b2706b6f91b946328928937a9d1c5090 (table test-table [id=b466f46a98d549a4877e3bd7eebff926]) successfully deleted
W20260927 02:44:47.134599 14863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:47.134837 14863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:47.134871 14863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:47.136056 14863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:47.136099 14863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:47.136142 14863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:47.137630 14863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44907
--tserver_master_addrs=127.0.98.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:47.137879 14863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:47.138091 14863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:47.143203 14873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.143327 14875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.143203 14872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:47.143438 14863 server_base.cc:1034] running on GCE node
I20260927 02:44:47.143637 14863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:47.143895 14863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:47.145069 14863 hybrid_clock.cc:648] HybridClock initialized: now 1790477087145056 us; error 29 us; skew 500 ppm
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:47.147502 14863 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:47.147933 14863 webserver.cc:460] Webserver started at http://127.0.98.2:44907/ using document root <none> and password file <none>
I20260927 02:44:47.148106 14863 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:47.148156 14863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:47.149595 14863 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:47.150209 14882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.150352 14863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:44:47.150430 14863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "f3f230c42067402aa64aebb864f08fae"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "6de0ffed9bb5e9440b98358b7e575175"
server_key_iv: "94649a62cfbdf8b76fdded52ddfd87ba"
server_key_version: "encryptionkey@0"
I20260927 02:44:47.150794 14863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:47.176946 14863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:47.179189 14863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:47.179311 14863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:47.179548 14863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:47.179976 14889 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:44:47.181494 14863 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:44:47.181552 14863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:44:47.181577 14863 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:44:47.182241 14863 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:44:47.182274 14863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:47.182356 14889 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Bootstrap starting.
I20260927 02:44:47.187980 14863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39485
I20260927 02:44:47.188120 14996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39485 every 8 connection(s)
I20260927 02:44:47.188376 14863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:47.192260 14889 log.cc:824] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:47.194823 14889 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:44:47.195117 14889 tablet_bootstrap.cc:492] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Bootstrap complete.
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:47.195909 14889 ts_tablet_manager.cc:1397] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260927 02:44:47.197073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 14863
I20260927 02:44:47.197328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44041
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46627
--webserver_port=36063
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:47.197191 14889 raft_consensus.cc:348] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.197520 14889 raft_consensus.cc:729] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f230c42067402aa64aebb864f08fae, State: Initialized, Role: FOLLOWER
I20260927 02:44:47.197659 14889 consensus_queue.cc:260] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.197954 14889 ts_tablet_manager.cc:1428] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260927 02:44:47.198844 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.189189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58945 (local address 127.0.98.62:46627)
0927 02:44:47.189272 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.189275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.189526 (+ 251us) server_negotiation.cc:407] Connection header received
0927 02:44:47.190235 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.190236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.190250 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.190307 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:47.191226 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.191772 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.193142 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.193358 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.193421 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.195818 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:47.195819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:47.195820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:47.195828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:47.197405 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.197864 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.197865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.197866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.197927 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.198190 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.198191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.198191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.198310 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:47.198369 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.198701 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.198779 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:44:47.199033 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.189088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:58945)
0927 02:44:47.189360 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.189416 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.190084 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.190382 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.190387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.190798 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:47.191120 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.191127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.191872 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.191873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.193026 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.193029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.193465 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.193466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.193607 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.194142 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:47.194153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:47.195727 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:47.197495 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.197500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.197502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.197770 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.197996 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.197996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.197997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.198119 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.198369 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:47.198376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:47.198552 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 02:44:47.198664 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 02:44:47.198833 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":101,"threads_started":1}
I20260927 02:44:47.199498 14997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46627
I20260927 02:44:47.199582 14997 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:47.199735 14997 heartbeater.cc:511] Master 127.0.98.62:46627 requested a full tablet report, sending...
I20260927 02:44:47.200224 14670 ts_manager.cc:194] Registered new tserver with Master: f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485)
I20260927 02:44:47.200799 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58945
I20260927 02:44:47.201562 14997 heartbeater.cc:503] Master 127.0.98.62:46627 was elected leader, sending a full tablet report...
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 02:44:47.208495 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.200501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39485 (local address 127.0.0.1:40652)
0927 02:44:47.200544 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.200554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.200587 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.200859 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.200861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.201018 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:47.201068 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.201070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.202102 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.202103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.202884 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.202887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.203491 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.203494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.203565 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.203917 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:47.203922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:47.205233 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:47.207410 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.207411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.207411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.207571 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.207789 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.207790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.207790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.207861 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.208173 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:47.208174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:47.208215 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:44:47.208277 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:44:47.208359 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:44:47.208534 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.200575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40652 (local address 127.0.98.2:39485)
0927 02:44:47.200711 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.200716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.200725 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:44:47.200755 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.200757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.200772 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.200835 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:47.201148 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.202047 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.202985 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.203417 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.203476 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.205332 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:47.205339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:47.205339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:47.205346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:47.207318 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.207644 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.207648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.207648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.207709 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.207924 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.207925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.207925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.208091 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:47.208167 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.208323 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.208436 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260927 02:44:47.208909 14931 tablet_service.cc:1514] Processing DeleteTablet for tablet b2706b6f91b946328928937a9d1c5090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40652
I20260927 02:44:47.209483 15004 tablet_replica.cc:330] stopping tablet replica
I20260927 02:44:47.209611 15004 raft_consensus.cc:2229] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:47.209683 15004 raft_consensus.cc:2258] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:47.209975 15004 ts_tablet_manager.cc:1905] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:47.211963 15004 ts_tablet_manager.cc:1918] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:47.212042 15004 log.cc:1196] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/b2706b6f91b946328928937a9d1c5090
I20260927 02:44:47.212289 15004 ts_tablet_manager.cc:1939] T b2706b6f91b946328928937a9d1c5090 P f3f230c42067402aa64aebb864f08fae: Deleting consensus metadata
I20260927 02:44:47.212859 14657 catalog_manager.cc:4841] TS f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485): tablet b2706b6f91b946328928937a9d1c5090 (table test-table [id=b466f46a98d549a4877e3bd7eebff926]) successfully deleted
W20260927 02:44:47.280223 15002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:47.280447 15002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:47.280469 15002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:47.281605 15002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:47.281647 15002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:47.281672 15002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:47.283085 15002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36063
--tserver_master_addrs=127.0.98.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:47.283303 15002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:47.283509 15002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:47.288511 15009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.288493 15012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.288491 15010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:47.288522 15002 server_base.cc:1034] running on GCE node
I20260927 02:44:47.288903 15002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:47.289099 15002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:47.290246 15002 hybrid_clock.cc:648] HybridClock initialized: now 1790477087290228 us; error 26 us; skew 500 ppm
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:47.292992 15002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:47.293393 15002 webserver.cc:460] Webserver started at http://127.0.98.3:36063/ using document root <none> and password file <none>
I20260927 02:44:47.293599 15002 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:47.293660 15002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:47.295094 15002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:47.295773 15019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.295938 15002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:47.296020 15002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "5ee5870f4e7541108d6b4075b7c84390"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "99d05f7310c76ada59af8aa43a9cac28"
server_key_iv: "4ae2602452359721e25d457a38070f67"
server_key_version: "encryptionkey@0"
I20260927 02:44:47.296368 15002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:47.309581 15002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:47.311852 15002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:47.312011 15002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:47.312266 15002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:47.312613 15002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:47.312647 15002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.312678 15002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:47.312696 15002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.318439 15002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44041
I20260927 02:44:47.318481 15132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44041 every 8 connection(s)
I20260927 02:44:47.318823 15002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:47.322443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 15002
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:44:47.328313 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.319553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37775 (local address 127.0.98.62:46627)
0927 02:44:47.319646 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.319648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.319822 (+ 174us) server_negotiation.cc:407] Connection header received
0927 02:44:47.320352 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.320353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.320366 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.320435 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:47.321379 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.321877 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.323008 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.323204 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.323259 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.325626 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:47.325627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:47.325628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:47.325635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:47.327021 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.327387 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.327388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.327388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.327443 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.327657 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.327657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.327658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.327768 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:47.327821 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.328084 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.328216 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 02:44:47.328462 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.319538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:37775)
0927 02:44:47.319690 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.319733 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.320249 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.320507 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.320511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.321077 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:47.321287 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.321292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.321957 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.321958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.322936 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.322939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.323271 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.323272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.323415 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.323898 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:47.323906 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:47.325543 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:47.327093 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.327100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.327102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.327321 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.327495 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:47.327496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:47.327496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:47.327602 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:47.327826 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:47.327832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:47.327965 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 02:44:47.328053 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 02:44:47.328216 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260927 02:44:47.328871 15133 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46627
I20260927 02:44:47.328965 15133 heartbeater.cc:464] Registering TS with master...
I20260927 02:44:47.329149 15133 heartbeater.cc:511] Master 127.0.98.62:46627 requested a full tablet report, sending...
I20260927 02:44:47.329296 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.322816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.0.98.62:46627)
0927 02:44:47.322925 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.322927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.323299 (+ 372us) server_negotiation.cc:407] Connection header received
0927 02:44:47.323316 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.323317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.323327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.323385 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:47.323889 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.324378 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.325508 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.325732 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.325783 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.326887 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:47.326888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:47.326889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:47.326895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:47.328240 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.328600 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.328601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.328601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.328653 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:47.328828 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:47.328829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:47.328829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:47.328932 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:47.329083 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.329138 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.329204 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260927 02:44:47.329525 14669 ts_manager.cc:194] Registered new tserver with Master: 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041)
I20260927 02:44:47.329865 14669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37775
I20260927 02:44:47.330436 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:44:47.332939 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.330766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.0.98.62:46627)
0927 02:44:47.330815 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.330817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.330857 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:44:47.330924 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.330925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.330957 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.331012 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:47.331100 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.331561 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.332254 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.332497 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.332544 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.332684 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:47.332776 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:47.332844 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.332862 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.332881 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:44:47.333189 14669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:44:47.334750 14669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:44:47.336732 14792 tablet_service.cc:1467] Processing CreateTablet for tablet 62ff838022ef4606acb59346937cfca9 (DEFAULT_TABLE table=test-table [id=ff4d29a0364d4da4a8c61c064529af77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:47.336789 14931 tablet_service.cc:1467] Processing CreateTablet for tablet 62ff838022ef4606acb59346937cfca9 (DEFAULT_TABLE table=test-table [id=ff4d29a0364d4da4a8c61c064529af77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:47.336925 14792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ff838022ef4606acb59346937cfca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:47.336972 14931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ff838022ef4606acb59346937cfca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:47.338176 14750 tablet_bootstrap.cc:492] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Bootstrap starting.
I20260927 02:44:47.338474 14889 tablet_bootstrap.cc:492] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: Bootstrap starting.
I20260927 02:44:47.338475 14750 tablet_bootstrap.cc:654] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:47.338738 14889 tablet_bootstrap.cc:654] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:47.339030 14750 tablet_bootstrap.cc:492] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: No bootstrap required, opened a new log
I20260927 02:44:47.339097 14750 ts_tablet_manager.cc:1397] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:47.339449 14750 raft_consensus.cc:348] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.339545 14889 tablet_bootstrap.cc:492] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: No bootstrap required, opened a new log
I20260927 02:44:47.339531 14750 raft_consensus.cc:374] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:47.339576 14750 raft_consensus.cc:729] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e160c6587c4537a375afe58f592510, State: Initialized, Role: FOLLOWER
I20260927 02:44:47.339600 14889 ts_tablet_manager.cc:1397] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:47.339622 14750 consensus_queue.cc:260] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.339712 14750 ts_tablet_manager.cc:1428] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:47.339748 14889 raft_consensus.cc:348] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.339811 14889 raft_consensus.cc:374] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:47.339851 14889 raft_consensus.cc:729] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f230c42067402aa64aebb864f08fae, State: Initialized, Role: FOLLOWER
I20260927 02:44:47.339902 14889 consensus_queue.cc:260] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.339984 14889 ts_tablet_manager.cc:1428] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.340420 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.336515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44041 (local address 127.0.0.1:40458)
0927 02:44:47.336548 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.336559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.336594 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.336999 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.337000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.337001 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:47.337041 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.337042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.338251 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.338252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.339267 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.339270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.340179 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.340181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.340243 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.340265 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:44:47.340293 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:44:47.340330 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 02:44:47.340715 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.336635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.0.98.3:44041)
0927 02:44:47.336821 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.336826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.336836 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:44:47.336868 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.336871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.336886 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.337109 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:47.337169 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.338136 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.339363 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.340087 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.340525 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.340595 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.340619 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.340657 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260927 02:44:47.341282 15067 tablet_service.cc:1467] Processing CreateTablet for tablet 62ff838022ef4606acb59346937cfca9 (DEFAULT_TABLE table=test-table [id=ff4d29a0364d4da4a8c61c064529af77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:44:47.341557 15067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ff838022ef4606acb59346937cfca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:44:47.344161 15142 tablet_bootstrap.cc:492] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: Bootstrap starting.
I20260927 02:44:47.344743 15142 tablet_bootstrap.cc:654] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: Neither blocks nor log segments found. Creating new log.
I20260927 02:44:47.345247 15142 log.cc:824] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:47.346014 15142 tablet_bootstrap.cc:492] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: No bootstrap required, opened a new log
I20260927 02:44:47.346069 15142 ts_tablet_manager.cc:1397] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:44:47.347363 15142 raft_consensus.cc:348] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.347505 15142 raft_consensus.cc:374] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:44:47.347527 15142 raft_consensus.cc:729] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee5870f4e7541108d6b4075b7c84390, State: Initialized, Role: FOLLOWER
I20260927 02:44:47.347628 15142 consensus_queue.cc:260] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.347885 15142 ts_tablet_manager.cc:1428] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:44:47.347939 15133 heartbeater.cc:503] Master 127.0.98.62:46627 was elected leader, sending a full tablet report...
I20260927 02:44:47.414644 15001 raft_consensus.cc:482] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:47.414805 15001 raft_consensus.cc:504] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.415102 15001 leader_election.cc:288] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041), 01e160c6587c4537a375afe58f592510 (127.0.98.1:44033)
I20260927 02:44:47.418198 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.415441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42107 (local address 127.0.98.1:44033)
0927 02:44:47.415483 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.415485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.415512 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:44:47.415540 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.415540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.415562 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.415607 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:47.415791 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.416321 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.417390 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.417862 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.417922 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.418011 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.418073 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.418117 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:44:47.418181 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.415428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38335 (local address 127.0.98.3:44041)
0927 02:44:47.415488 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.415490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.415605 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:44:47.415686 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.415687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.415697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.415744 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:44:47.415897 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.416371 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.417395 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.417802 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.417864 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.417876 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.418060 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.418112 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:44:47.418200 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.415320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44033 (local address 127.0.98.2:42107)
0927 02:44:47.415375 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.415389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.415462 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.415620 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.415621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.415623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:47.415721 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.415722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.416428 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.416429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.417291 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.417294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.417935 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.417936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.418006 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.418010 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:44:47.418066 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:44:47.418109 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":70}
I20260927 02:44:47.418210 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.415337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44041 (local address 127.0.98.2:38335)
0927 02:44:47.415523 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.415534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.415623 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.415757 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:47.415758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:47.415760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:44:47.415837 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.415838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.416461 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.416462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.417316 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:47.417318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.417886 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.417888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:47.417966 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.417987 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:44:47.418034 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:44:47.418076 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
I20260927 02:44:47.418550 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ff838022ef4606acb59346937cfca9" candidate_uuid: "f3f230c42067402aa64aebb864f08fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e160c6587c4537a375afe58f592510" is_pre_election: true
I20260927 02:44:47.418730 14812 raft_consensus.cc:2454] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3f230c42067402aa64aebb864f08fae in term 0.
I20260927 02:44:47.418885 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ff838022ef4606acb59346937cfca9" candidate_uuid: "f3f230c42067402aa64aebb864f08fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee5870f4e7541108d6b4075b7c84390" is_pre_election: true
I20260927 02:44:47.418962 14884 leader_election.cc:302] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e160c6587c4537a375afe58f592510, f3f230c42067402aa64aebb864f08fae; no voters:
I20260927 02:44:47.419061 15087 raft_consensus.cc:2454] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3f230c42067402aa64aebb864f08fae in term 0.
I20260927 02:44:47.419080 15001 raft_consensus.cc:2781] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:44:47.419119 15001 raft_consensus.cc:482] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:44:47.419148 15001 raft_consensus.cc:3037] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:47.419848 15001 raft_consensus.cc:504] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.419981 15001 leader_election.cc:288] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [CANDIDATE]: Term 1 election: Requested vote from peers 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041), 01e160c6587c4537a375afe58f592510 (127.0.98.1:44033)
I20260927 02:44:47.420169 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ff838022ef4606acb59346937cfca9" candidate_uuid: "f3f230c42067402aa64aebb864f08fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e160c6587c4537a375afe58f592510"
I20260927 02:44:47.420224 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ff838022ef4606acb59346937cfca9" candidate_uuid: "f3f230c42067402aa64aebb864f08fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee5870f4e7541108d6b4075b7c84390"
I20260927 02:44:47.420269 14812 raft_consensus.cc:3037] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:47.420293 15087 raft_consensus.cc:3037] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:44:47.421162 15087 raft_consensus.cc:2454] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3f230c42067402aa64aebb864f08fae in term 1.
I20260927 02:44:47.421236 14812 raft_consensus.cc:2454] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3f230c42067402aa64aebb864f08fae in term 1.
I20260927 02:44:47.421470 14884 leader_election.cc:302] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e160c6587c4537a375afe58f592510, f3f230c42067402aa64aebb864f08fae; no voters:
I20260927 02:44:47.421643 15001 raft_consensus.cc:2781] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:44:47.421835 15001 raft_consensus.cc:686] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 1 LEADER]: Becoming Leader. State: Replica: f3f230c42067402aa64aebb864f08fae, State: Running, Role: LEADER
I20260927 02:44:47.421904 15001 consensus_queue.cc:237] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.422760 14669 catalog_manager.cc:5495] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae reported cstate change: term changed from 0 to 1, leader changed from <none> to f3f230c42067402aa64aebb864f08fae (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f3f230c42067402aa64aebb864f08fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } health_report { overall_health: HEALTHY } } }
I20260927 02:44:47.434317 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.431988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.0.98.2:39485)
0927 02:44:47.432049 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:47.432051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:47.432059 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:44:47.432097 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:47.432098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:47.432116 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:47.432175 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:44:47.432288 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.432833 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.433558 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:47.433772 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:47.433826 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:47.433990 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:44:47.434104 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:44:47.434166 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:47.434212 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:44:47.434236 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:44:47.435801 15087 raft_consensus.cc:1261] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [term 1 FOLLOWER]: Refusing update from remote peer f3f230c42067402aa64aebb864f08fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:47.435838 14812 raft_consensus.cc:1261] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Refusing update from remote peer f3f230c42067402aa64aebb864f08fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:44:47.436192 15001 consensus_queue.cc:1035] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:47.436344 15146 consensus_queue.cc:1035] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:44:47.439311 15151 mvcc.cc:205] Tried to move back new op lower bound from 7333794150134693888 to 7333794150080905216. Current Snapshot: MvccSnapshot[applied={T|T < 7333794150134693888}]
I20260927 02:44:47.440850 14669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:47.440943 14669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:44:47.441643 14669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Sending DeleteTablet for 3 replicas of tablet 62ff838022ef4606acb59346937cfca9
I20260927 02:44:47.441927 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ff838022ef4606acb59346937cfca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49046
I20260927 02:44:47.441984 14931 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ff838022ef4606acb59346937cfca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40652
I20260927 02:44:47.442028 14867 tablet_replica.cc:330] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: stopping tablet replica
I20260927 02:44:47.442075 14867 raft_consensus.cc:2229] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:47.442095 15004 tablet_replica.cc:330] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: stopping tablet replica
I20260927 02:44:47.442116 14867 raft_consensus.cc:2258] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:47.442075 15067 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ff838022ef4606acb59346937cfca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:44:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40458
I20260927 02:44:47.442135 15004 raft_consensus.cc:2229] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:44:47.442320 15004 raft_consensus.cc:2258] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:44:47.442363 15159 tablet_replica.cc:330] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: stopping tablet replica
I20260927 02:44:47.442540 15159 raft_consensus.cc:2229] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:44:47.442571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 14708
I20260927 02:44:47.442734 15159 raft_consensus.cc:2258] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:44:47.447825 14655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44033 (error 108)
I20260927 02:44:47.448144 15004 ts_tablet_manager.cc:1905] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:47.448419 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.448275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44033 (local address 127.0.0.1:49048)
0927 02:44:47.448324 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.448361 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:44:47.448412 15159 ts_tablet_manager.cc:1905] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:44:47.448539 14655 catalog_manager.cc:4584] TS 01e160c6587c4537a375afe58f592510 (127.0.98.1:44033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62ff838022ef4606acb59346937cfca9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44033: connect: Connection refused (error 111)
I20260927 02:44:47.449168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 14863
I20260927 02:44:47.450740 15159 ts_tablet_manager.cc:1918] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:47.450827 15159 log.cc:1196] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/62ff838022ef4606acb59346937cfca9
I20260927 02:44:47.451114 15159 ts_tablet_manager.cc:1939] T 62ff838022ef4606acb59346937cfca9 P 5ee5870f4e7541108d6b4075b7c84390: Deleting consensus metadata
I20260927 02:44:47.451541 14654 catalog_manager.cc:4841] TS 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041): tablet 62ff838022ef4606acb59346937cfca9 (table test-table [id=ff4d29a0364d4da4a8c61c064529af77]) successfully deleted
I20260927 02:44:47.453948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 15002
I20260927 02:44:47.453979 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.453886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39485 (local address 127.0.0.1:40674)
0927 02:44:47.453926 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.453952 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 02:44:47.458127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 14637
I20260927 02:44:47.462612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46627
--webserver_interface=127.0.98.62
--webserver_port=34575
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:47.543493 15161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:47.543753 15161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:47.543778 15161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:47.545102 15161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:44:47.545162 15161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:47.545176 15161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:47.545189 15161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:44:47.545200 15161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:44:47.546836 15161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:47.547106 15161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:47.547348 15161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:47.552069 15166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.552069 15167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.552217 15169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:47.552525 15161 server_base.cc:1034] running on GCE node
I20260927 02:44:47.552780 15161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:47.553066 15161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:47.554256 15161 hybrid_clock.cc:648] HybridClock initialized: now 1790477087554238 us; error 34 us; skew 500 ppm
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:47.556823 15161 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:44:47.557263 15161 webserver.cc:460] Webserver started at http://127.0.98.62:34575/ using document root <none> and password file <none>
I20260927 02:44:47.557448 15161 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:47.557511 15161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:47.558888 15161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:47.559731 15176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.559962 15161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:47.560057 15161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "4d305b94633148258630996f49fa8328"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "659dee896b1ae459497ac9ace5220afc"
server_key_iv: "fd73ec01783a9586b323b219be767f03"
server_key_version: "encryptionkey@0"
I20260927 02:44:47.560509 15161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:47.568140 15161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:47.570263 15161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:47.570384 15161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:47.574204 15161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46627
I20260927 02:44:47.574246 15228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46627 every 8 connection(s)
I20260927 02:44:47.574591 15161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:44:47.576359 15229 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:44:47.577117 15229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Bootstrap starting.
I20260927 02:44:47.577975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 15161
I20260927 02:44:47.578274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44033
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46627
--webserver_port=45821
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:47.585836 15229 log.cc:824] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:47.588248 15229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:44:47.588572 15229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328: Bootstrap complete.
I20260927 02:44:47.590952 15229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:47.591269 15229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d305b94633148258630996f49fa8328, State: Initialized, Role: FOLLOWER
I20260927 02:44:47.591427 15229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:47.591493 15229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:44:47.591535 15229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:44:47.591581 15229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:44:47.592680 15229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:47.592774 15229 leader_election.cc:302] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d305b94633148258630996f49fa8328; no voters:
I20260927 02:44:47.592934 15229 leader_election.cc:288] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:44:47.592988 15233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:44:47.593111 15233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [term 3 LEADER]: Becoming Leader. State: Replica: 4d305b94633148258630996f49fa8328, State: Running, Role: LEADER
I20260927 02:44:47.593199 15233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } }
I20260927 02:44:47.593225 15229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:44:47.593794 15234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d305b94633148258630996f49fa8328. Latest consensus state: current_term: 3 leader_uuid: "4d305b94633148258630996f49fa8328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } } }
I20260927 02:44:47.593873 15234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:47.594184 15241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:44:47.594210 15235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4d305b94633148258630996f49fa8328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d305b94633148258630996f49fa8328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46627 } } }
I20260927 02:44:47.594269 15235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d305b94633148258630996f49fa8328 [sys.catalog]: This master's current role is: LEADER
I20260927 02:44:47.594967 15241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:44:47.595072 15241 catalog_manager.cc:1259] Loaded cluster ID: 713d31fd03c341f6a6505193e185d5ed
I20260927 02:44:47.595101 15241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 02:44:47.595361 15241 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f01f573a9ad google::LogMessage::Fail() at ??:0
@ 0x7f01f573d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f01f573a4de google::LogMessage::Flush() at ??:0
@ 0x7f01f573ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f01fd65f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f01fd6688b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f01f76080d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f01f75fd954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f01f84616db start_thread at ??:0
@ 0x7f01f423861f clone at ??:0
W20260927 02:44:47.665804 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:47.666013 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:47.666046 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:47.667236 15231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:47.667279 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:47.667313 15231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:44:47.668852 15231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45821
--tserver_master_addrs=127.0.98.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:47.669096 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:47.669304 15231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:47.674175 15265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.674247 15266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.674183 15268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:47.674516 15231 server_base.cc:1034] running on GCE node
I20260927 02:44:47.674712 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:47.674947 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:47.676097 15231 hybrid_clock.cc:648] HybridClock initialized: now 1790477087676082 us; error 32 us; skew 500 ppm
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:47.678735 15231 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:44:47.679173 15231 webserver.cc:460] Webserver started at http://127.0.98.1:45821/ using document root <none> and password file <none>
I20260927 02:44:47.679394 15231 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:47.679445 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:47.680920 15231 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:47.681669 15275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.681846 15231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:47.681936 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "01e160c6587c4537a375afe58f592510"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "cd0055cee0f92559dd96a2335d0f7595"
server_key_iv: "7f6787f9797eafff28701d3ea8d8ef10"
server_key_version: "encryptionkey@0"
I20260927 02:44:47.682304 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:47.717203 15231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:47.719527 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:47.719691 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:47.719965 15231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:47.720455 15282 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:44:47.722421 15231 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:44:47.722482 15231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:44:47.722520 15231 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:44:47.723237 15231 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:44:47.723274 15231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:44:47.723376 15282 tablet_bootstrap.cc:492] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Bootstrap starting.
I20260927 02:44:47.730456 15231 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44033
I20260927 02:44:47.730594 15389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44033 every 8 connection(s)
I20260927 02:44:47.730894 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:44:47.733824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 15231
I20260927 02:44:47.733930 15282 log.cc:824] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Log is configured to *not* fsync() on all Append() calls
I20260927 02:44:47.734102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39485
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46627
--webserver_port=44907
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:44:47.736161 15282 tablet_bootstrap.cc:492] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:44:47.736461 15282 tablet_bootstrap.cc:492] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Bootstrap complete.
I20260927 02:44:47.737205 15282 ts_tablet_manager.cc:1397] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260927 02:44:47.738276 15282 raft_consensus.cc:348] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.738500 15282 raft_consensus.cc:729] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e160c6587c4537a375afe58f592510, State: Initialized, Role: FOLLOWER
I20260927 02:44:47.738642 15282 consensus_queue.cc:260] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:47.738922 15282 ts_tablet_manager.cc:1428] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260927 02:44:47.797857 15392 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:44:47.731595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:60375)
0927 02:44:47.731790 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.731835 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:47.732299 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:47.797653 (+ 65354us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
W20260927 02:44:47.798116 15390 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46627 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46627: Client connection negotiation failed: client connection to 127.0.98.62:46627: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:44:47.798431 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.798324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:41279)
0927 02:44:47.798371 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.798409 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:44:47.798750 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.798661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:52741)
0927 02:44:47.798704 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.798728 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:44:47.798867 15390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:44:47.818437 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:47.818653 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:47.818678 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:47.819856 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:47.819896 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:47.819922 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:44:47.821481 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44907
--tserver_master_addrs=127.0.98.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:47.821686 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:47.821872 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:47.826865 15400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.826872 15403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.826928 15401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:47.827162 15394 server_base.cc:1034] running on GCE node
I20260927 02:44:47.827381 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:47.827642 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:47.828804 15394 hybrid_clock.cc:648] HybridClock initialized: now 1790477087828772 us; error 45 us; skew 500 ppm
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:47.831357 15394 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:44:47.831768 15394 webserver.cc:460] Webserver started at http://127.0.98.2:44907/ using document root <none> and password file <none>
I20260927 02:44:47.831947 15394 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:47.831987 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:47.833449 15394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:47.834122 15410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.834343 15394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:44:47.834442 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "f3f230c42067402aa64aebb864f08fae"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "6de0ffed9bb5e9440b98358b7e575175"
server_key_iv: "94649a62cfbdf8b76fdded52ddfd87ba"
server_key_version: "encryptionkey@0"
I20260927 02:44:47.834832 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:47.849330 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:47.851203 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:47.851348 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:47.851555 15394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:47.851984 15417 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 02:44:47.853425 15417 ts_tablet_manager.cc:1852] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 02:44:47.853730 15417 ts_tablet_manager.cc:1905] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:44:47.855779 15417 ts_tablet_manager.cc:1918] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:44:47.855851 15417 log.cc:1196] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/62ff838022ef4606acb59346937cfca9
I20260927 02:44:47.856096 15417 ts_tablet_manager.cc:1939] T 62ff838022ef4606acb59346937cfca9 P f3f230c42067402aa64aebb864f08fae: Deleting consensus metadata
I20260927 02:44:47.856305 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 02:44:47.856348 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 02:44:47.856381 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:47.856428 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.862126 15394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39485
I20260927 02:44:47.862255 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39485 every 8 connection(s)
I20260927 02:44:47.862470 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:44:47.863454 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.863073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:39571)
0927 02:44:47.863239 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.863316 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260927 02:44:47.863693 15525 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46627 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46627: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
I20260927 02:44:47.863962 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.863867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:34437)
0927 02:44:47.863909 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.863937 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:44:47.864305 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.864222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:52417)
0927 02:44:47.864258 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.864282 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:44:47.864414 15525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:44:47.870204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 15394
I20260927 02:44:47.870468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44041
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46627
--webserver_port=36063
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:44:47.951292 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:44:47.951501 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:44:47.951524 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:44:47.952750 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:44:47.952796 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:44:47.952821 15528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:44:47.954282 15528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36063
--tserver_master_addrs=127.0.98.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:44:47.954510 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:44:47.954731 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:44:47.959556 15533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.959560 15536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:44:47.959605 15534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:44:47.959868 15528 server_base.cc:1034] running on GCE node
I20260927 02:44:47.960036 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:44:47.960294 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:44:47.961441 15528 hybrid_clock.cc:648] HybridClock initialized: now 1790477087961433 us; error 41 us; skew 500 ppm
Sep 27 02:44:47 dist-test-slave-44kr krb5kdc[14093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:44:47.963958 15528 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:44:47.964372 15528 webserver.cc:460] Webserver started at http://127.0.98.3:36063/ using document root <none> and password file <none>
I20260927 02:44:47.964614 15528 fs_manager.cc:359] Metadata directory not provided
I20260927 02:44:47.964661 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:44:47.966038 15528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:47.966748 15543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.966919 15528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:44:47.967016 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "5ee5870f4e7541108d6b4075b7c84390"
format_stamp: "Formatted at 2026-09-27 02:44:46 on dist-test-slave-44kr"
server_key: "99d05f7310c76ada59af8aa43a9cac28"
server_key_iv: "4ae2602452359721e25d457a38070f67"
server_key_version: "encryptionkey@0"
I20260927 02:44:47.967378 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:44:47.981105 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:44:47.983091 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:44:47.983207 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:44:47.983435 15528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:44:47.983752 15528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:44:47.983788 15528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.983810 15528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:44:47.983834 15528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:44:47.989543 15528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44041
I20260927 02:44:47.989609 15656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44041 every 8 connection(s)
I20260927 02:44:47.989935 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:44:47.990973 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.990625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:33299)
0927 02:44:47.990762 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.990833 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
W20260927 02:44:47.991357 15657 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46627 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46627: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
I20260927 02:44:47.991600 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.991522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:53663)
0927 02:44:47.991559 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.991583 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:44:47.991807 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:47.991750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:39383)
0927 02:44:47.991779 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:47.991796 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 02:44:47.991886 15657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:44:47.995540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 15528
I20260927 02:44:48.799619 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.799320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:43993)
0927 02:44:48.799475 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:48.799524 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260927 02:44:48.865016 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.864797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:56841)
0927 02:44:48.864934 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:48.864977 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260927 02:44:48.885807 15663 raft_consensus.cc:482] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:48.885931 15663 raft_consensus.cc:504] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:48.886240 15663 leader_election.cc:288] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041), f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485)
Sep 27 02:44:48 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 02:44:48 dist-test-slave-44kr krb5kdc[14093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 02:44:48.896234 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.886561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53581 (local address 127.0.98.2:39485)
0927 02:44:48.886748 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:48.886762 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:48.886786 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:44:48.887322 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:48.887325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:48.887365 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:48.887495 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:48.888155 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.889048 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.890044 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.890423 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.890525 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.893294 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:48.893297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:48.893298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:48.893307 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:48.894765 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:48.895138 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:48.895140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:48.895140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:48.895222 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:48.895441 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:48.895441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:48.895442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:48.895665 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:48.895731 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:48.895998 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 02:44:48.896120 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260927 02:44:48.896235 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.886587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35547 (local address 127.0.98.3:44041)
0927 02:44:48.886751 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:44:48.886762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:44:48.886788 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:44:48.887249 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:44:48.887251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:44:48.887306 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:44:48.887419 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:44:48.888163 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.889057 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.890042 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.890423 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.890525 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.892696 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:44:48.892700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:44:48.892701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:44:48.892711 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:44:48.894565 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:48.895073 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:48.895076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:48.895077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:48.895240 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:44:48.895440 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:44:48.895441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:44:48.895442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:44:48.895674 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:44:48.895750 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:44:48.896004 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 02:44:48.896123 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":54,"threads_started":1}
I20260927 02:44:48.896235 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.886419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39485 (local address 127.0.98.1:53581)
0927 02:44:48.886520 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:48.886528 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:48.886569 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:48.887498 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:48.887500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:48.887863 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:48.888053 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.888058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.889139 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.889140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.889966 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.889968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.890501 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.890502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.890613 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.891119 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:48.891129 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:48.892983 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:48.894869 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:48.894870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:48.894870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:48.895058 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:48.895283 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:48.895284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:48.895285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:48.895382 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:48.895730 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:48.895736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:48.895853 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 02:44:48.895939 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:44:48.896133 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":76,"thread_start_us":66,"threads_started":1}
I20260927 02:44:48.896420 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.886402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44041 (local address 127.0.98.1:35547)
0927 02:44:48.886460 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:48.886472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:44:48.886522 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:44:48.887484 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:44:48.887488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:44:48.887862 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:44:48.888053 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.888058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.889149 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.889150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.889965 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:44:48.889967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:44:48.890500 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:44:48.890501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:44:48.890615 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:44:48.891156 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:44:48.891160 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:44:48.892559 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:44:48.894661 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:48.894668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:48.894671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:48.894981 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:48.895311 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:44:48.895312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:44:48.895312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:44:48.895382 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:44:48.895744 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:44:48.895745 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:44:48.895852 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 02:44:48.895938 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:44:48.896207 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":160}
W20260927 02:44:48.897617 15276 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
W20260927 02:44:48.897688 15279 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
I20260927 02:44:48.897747 15279 leader_election.cc:302] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01e160c6587c4537a375afe58f592510; no voters: 5ee5870f4e7541108d6b4075b7c84390, f3f230c42067402aa64aebb864f08fae
I20260927 02:44:48.897842 15663 raft_consensus.cc:2726] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:44:48.992443 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:48.992210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:44969)
0927 02:44:48.992326 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:48.992373 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260927 02:44:49.800462 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:49.800163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:37973)
0927 02:44:49.800319 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:49.800362 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260927 02:44:49.865753 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:49.865519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:46307)
0927 02:44:49.865660 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:49.865705 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
I20260927 02:44:49.993093 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:49.992889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:48575)
0927 02:44:49.993018 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:49.993056 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260927 02:44:50.801241 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:50.800966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:43691)
0927 02:44:50.801143 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:50.801192 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260927 02:44:50.866576 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:50.866322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:58291)
0927 02:44:50.866481 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:50.866531 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260927 02:44:50.993829 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:50.993561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:48897)
0927 02:44:50.993734 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:50.993782 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260927 02:44:51.133705 15674 raft_consensus.cc:482] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:51.133823 15674 raft_consensus.cc:504] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:51.133965 15674 leader_election.cc:288] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041), f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485)
W20260927 02:44:51.134405 15279 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
W20260927 02:44:51.134498 15276 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
I20260927 02:44:51.134605 15276 leader_election.cc:302] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01e160c6587c4537a375afe58f592510; no voters: 5ee5870f4e7541108d6b4075b7c84390, f3f230c42067402aa64aebb864f08fae
I20260927 02:44:51.134681 15674 raft_consensus.cc:2726] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:44:51.802116 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:51.801830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:36445)
0927 02:44:51.801969 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:51.802008 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260927 02:44:51.867264 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:51.867045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:60451)
0927 02:44:51.867180 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:51.867219 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260927 02:44:51.994529 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:51.994304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:52071)
0927 02:44:51.994440 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:51.994484 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260927 02:44:52.802969 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:52.802758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:56493)
0927 02:44:52.802887 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:52.802919 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260927 02:44:52.867997 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:52.867758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:45363)
0927 02:44:52.867904 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:52.867946 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260927 02:44:52.995208 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:52.994994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:44951)
0927 02:44:52.995127 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:52.995164 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260927 02:44:53.608302 15681 raft_consensus.cc:482] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:53.608419 15681 raft_consensus.cc:504] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:53.608624 15681 leader_election.cc:288] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041), f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485)
W20260927 02:44:53.608971 15276 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
W20260927 02:44:53.609047 15279 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
I20260927 02:44:53.609117 15279 leader_election.cc:302] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01e160c6587c4537a375afe58f592510; no voters: 5ee5870f4e7541108d6b4075b7c84390, f3f230c42067402aa64aebb864f08fae
I20260927 02:44:53.609201 15681 raft_consensus.cc:2726] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:44:53.803788 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:53.803527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:45363)
0927 02:44:53.803678 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:53.803736 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260927 02:44:53.868795 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:53.868565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:48957)
0927 02:44:53.868706 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:53.868750 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260927 02:44:53.995981 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:53.995721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:49849)
0927 02:44:53.995866 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:53.995914 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
I20260927 02:44:54.804646 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:54.804421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:55705)
0927 02:44:54.804556 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:54.804598 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260927 02:44:54.869517 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:54.869295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:48039)
0927 02:44:54.869429 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:54.869472 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260927 02:44:54.996773 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:54.996558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:42955)
0927 02:44:54.996688 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:54.996729 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260927 02:44:55.805433 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.805151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:37295)
0927 02:44:55.805336 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:55.805386 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260927 02:44:55.870205 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.869973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:56309)
0927 02:44:55.870108 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:55.870156 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260927 02:44:55.997467 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:55.997240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:38781)
0927 02:44:55.997374 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:55.997421 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260927 02:44:56.061765 15691 raft_consensus.cc:482] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:44:56.061865 15691 raft_consensus.cc:504] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:44:56.062043 15691 leader_election.cc:288] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041), f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485)
W20260927 02:44:56.062556 15279 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
W20260927 02:44:56.062660 15276 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
I20260927 02:44:56.062727 15276 leader_election.cc:302] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01e160c6587c4537a375afe58f592510; no voters: 5ee5870f4e7541108d6b4075b7c84390, f3f230c42067402aa64aebb864f08fae
I20260927 02:44:56.062830 15691 raft_consensus.cc:2726] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:44:56.806298 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:56.806012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:34423)
0927 02:44:56.806192 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:56.806240 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260927 02:44:56.870898 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:56.870658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:46201)
0927 02:44:56.870797 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:56.870852 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260927 02:44:56.998111 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:56.997904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:34643)
0927 02:44:56.998028 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:56.998068 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260927 02:44:57.807143 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:57.806923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:52161)
0927 02:44:57.807050 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:57.807095 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260927 02:44:57.871701 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:57.871414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:52723)
0927 02:44:57.871610 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:57.871647 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260927 02:44:57.998816 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:57.998601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:56785)
0927 02:44:57.998729 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:57.998768 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260927 02:44:58.807898 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.807671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:37349)
0927 02:44:58.807810 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.807850 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260927 02:44:58.872560 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.872243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:41289)
0927 02:44:58.872457 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.872507 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260927 02:44:58.999485 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:58.999260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:53299)
0927 02:44:58.999398 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:58.999440 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260927 02:44:59.808765 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:59.808497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:58041)
0927 02:44:59.808670 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:59.808717 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260927 02:44:59.873268 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:59.873039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:44823)
0927 02:44:59.873179 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:44:59.873223 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260927 02:45:00.000275 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:44:59.999978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:57689)
0927 02:45:00.000174 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:00.000228 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":65,"threads_started":1}
I20260927 02:45:00.809476 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:00.809252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:46963)
0927 02:45:00.809383 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:00.809429 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260927 02:45:00.873962 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:00.873760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:59657)
0927 02:45:00.873877 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:00.873917 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260927 02:45:01.001107 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:01.000860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:54045)
0927 02:45:01.001005 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:01.001057 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260927 02:45:01.806538 15707 raft_consensus.cc:482] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:45:01.806668 15707 raft_consensus.cc:504] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee5870f4e7541108d6b4075b7c84390" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44041 } } peers { permanent_uuid: "01e160c6587c4537a375afe58f592510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44033 } } peers { permanent_uuid: "f3f230c42067402aa64aebb864f08fae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39485 } }
I20260927 02:45:01.806814 15707 leader_election.cc:288] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041), f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485)
W20260927 02:45:01.807246 15279 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3f230c42067402aa64aebb864f08fae (127.0.98.2:39485): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
W20260927 02:45:01.807391 15276 leader_election.cc:341] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ee5870f4e7541108d6b4075b7c84390 (127.0.98.3:44041): Not found: Tablet not found: 62ff838022ef4606acb59346937cfca9
I20260927 02:45:01.807457 15276 leader_election.cc:302] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01e160c6587c4537a375afe58f592510; no voters: 5ee5870f4e7541108d6b4075b7c84390, f3f230c42067402aa64aebb864f08fae
I20260927 02:45:01.807518 15707 raft_consensus.cc:2726] T 62ff838022ef4606acb59346937cfca9 P 01e160c6587c4537a375afe58f592510 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:45:01.810159 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:01.809903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:56495)
0927 02:45:01.810034 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:01.810068 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260927 02:45:01.874670 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:01.874430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:46801)
0927 02:45:01.874579 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:01.874623 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260927 02:45:02.001791 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:02.001582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:47315)
0927 02:45:02.001704 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:02.001743 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260927 02:45:02.811002 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:02.810703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.1:36593)
0927 02:45:02.810894 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:02.810949 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260927 02:45:02.875667 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:02.875402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.2:56135)
0927 02:45:02.875559 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:02.875618 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260927 02:45:03.002588 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:03.002364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46627 (local address 127.0.98.3:58055)
0927 02:45:03.002496 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:03.002540 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260927 02:45:03.005722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 15231
I20260927 02:45:03.009737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 15394
I20260927 02:45:03.013484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 15528
2026-09-27T02:45:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18074 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:45:03 dist-test-slave-44kr krb5kdc[15716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:45:03 dist-test-slave-44kr krb5kdc[15716](info): set up 2 sockets
Sep 27 02:45:03 dist-test-slave-44kr krb5kdc[15716](info): commencing operation
krb5kdc: starting...
W20260927 02:45:03.972175 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:45:04 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:45:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:45:04.092278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:45:04.173520 15729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:04.173736 15729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:04.173766 15729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:04.174935 15729 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:45:04.174975 15729 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:45:04.174991 15729 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:45:04.175006 15729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:45:04.175022 15729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:04.175040 15729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:04.175060 15729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:45:04.175091 15729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:45:04.176594 15729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:04.176831 15729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:04.177040 15729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:04.182036 15734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:04.182056 15735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:04.182152 15729 server_base.cc:1034] running on GCE node
W20260927 02:45:04.182204 15737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:04.182613 15729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:04.182884 15729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:04.184072 15729 hybrid_clock.cc:648] HybridClock initialized: now 1790477104184054 us; error 33 us; skew 500 ppm
Sep 27 02:45:04 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:04.186689 15729 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:45:04.187132 15729 webserver.cc:460] Webserver started at http://127.0.98.62:34531/ using document root <none> and password file <none>
I20260927 02:45:04.187390 15729 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:04.187441 15729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:04.187563 15729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:45:04.188685 15729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "54557d141cc74e43bab5bb488bb58ced"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "e813bba08ed1d2fc2f9af81045e0ea92"
server_key_iv: "3a369018bc3b1384f55a13a8f4c128f7"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.189021 15729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "54557d141cc74e43bab5bb488bb58ced"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "e813bba08ed1d2fc2f9af81045e0ea92"
server_key_iv: "3a369018bc3b1384f55a13a8f4c128f7"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.190330 15729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:45:04.191013 15744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.191187 15729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:45:04.191310 15729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "54557d141cc74e43bab5bb488bb58ced"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "e813bba08ed1d2fc2f9af81045e0ea92"
server_key_iv: "3a369018bc3b1384f55a13a8f4c128f7"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.191378 15729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:04.212175 15729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:04.214463 15729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:04.214610 15729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:04.218499 15729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38417
I20260927 02:45:04.218544 15796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38417 every 8 connection(s)
I20260927 02:45:04.218878 15729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:45:04.219760 15797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:04.222033 15797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Bootstrap starting.
I20260927 02:45:04.222625 15797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:04.222899 15797 log.cc:824] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:04.223536 15797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: No bootstrap required, opened a new log
I20260927 02:45:04.225096 15797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:04.225237 15797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:04.225261 15797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54557d141cc74e43bab5bb488bb58ced, State: Initialized, Role: FOLLOWER
I20260927 02:45:04.225404 15797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:04.225466 15797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:45:04.225488 15797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:45:04.225536 15797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:04.226019 15797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:04.226104 15797 leader_election.cc:302] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54557d141cc74e43bab5bb488bb58ced; no voters:
I20260927 02:45:04.226255 15797 leader_election.cc:288] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:45:04.226297 15800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:45:04.226454 15800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 1 LEADER]: Becoming Leader. State: Replica: 54557d141cc74e43bab5bb488bb58ced, State: Running, Role: LEADER
I20260927 02:45:04.226480 15797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:45:04.226540 15800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:04.227147 15801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54557d141cc74e43bab5bb488bb58ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260927 02:45:04.227237 15801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: This master's current role is: LEADER
I20260927 02:45:04.227524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 15729
I20260927 02:45:04.227607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:45:04.227855 392 external_mini_cluster.cc:1428] Setting key c239918aa4fbf8d605b0d23a6fcac0b8
I20260927 02:45:04.227933 15802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54557d141cc74e43bab5bb488bb58ced. Latest consensus state: current_term: 1 leader_uuid: "54557d141cc74e43bab5bb488bb58ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260927 02:45:04.228440 15802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: This master's current role is: LEADER
W20260927 02:45:04.228614 15815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:45:04.228668 15815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:45:04.228724 15809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:45:04.228916 15809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:45:04.230724 15809 catalog_manager.cc:1347] Generated new cluster ID: 2df93773bcaa404090f78fb718754a8e
I20260927 02:45:04.230769 15809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:45:04 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:04.237488 15809 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:45:04.238117 15809 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:45:04.241173 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.228870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.0.98.62:38417)
0927 02:45:04.229897 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.229906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.229939 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:45:04.230058 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.230065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.230135 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.230289 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:04.231119 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.232648 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.234047 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.234820 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.236603 (+ 1783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.237684 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:04.237688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:04.237688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:04.237701 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:04.239584 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.239972 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.239975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.239975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.240074 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.240265 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.240266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.240266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.240536 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:04.240674 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.240791 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.240933 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":980,"thread_start_us":88,"threads_started":1}
I20260927 02:45:04.241976 15809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Generated new TSK 0
I20260927 02:45:04.242120 15809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:45:04.270814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:45:04.352413 15823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:04.352648 15823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:04.352671 15823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:04.353875 15823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:04.353921 15823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:04.353947 15823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:45:04.355476 15823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:04.355698 15823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:04.355932 15823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:04.360711 15829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:04.360769 15831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:04.360908 15828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:04.360880 15823 server_base.cc:1034] running on GCE node
I20260927 02:45:04.361121 15823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:04.361387 15823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:04.362564 15823 hybrid_clock.cc:648] HybridClock initialized: now 1790477104362549 us; error 30 us; skew 500 ppm
Sep 27 02:45:04 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:04.365140 15823 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:45:04.365633 15823 webserver.cc:460] Webserver started at http://127.0.98.1:43753/ using document root <none> and password file <none>
I20260927 02:45:04.365830 15823 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:04.365872 15823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:04.365994 15823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:45:04.367038 15823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "93fedfbe449346cd89d1a12daf219cb5"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "f6e3b36cc39c952b1e8407a0783bddef"
server_key_iv: "896628e08b3bd800971ea86e7260afb4"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.367375 15823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "93fedfbe449346cd89d1a12daf219cb5"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "f6e3b36cc39c952b1e8407a0783bddef"
server_key_iv: "896628e08b3bd800971ea86e7260afb4"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.368724 15823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:45:04.369480 15838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.369649 15823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:04.369732 15823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "93fedfbe449346cd89d1a12daf219cb5"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "f6e3b36cc39c952b1e8407a0783bddef"
server_key_iv: "896628e08b3bd800971ea86e7260afb4"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.369794 15823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:04.383072 15823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:04.385432 15823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:04.385581 15823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:04.385807 15823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:04.386101 15823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:45:04.386137 15823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.386164 15823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:45:04.386189 15823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.392072 15823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43911
I20260927 02:45:04.392127 15951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43911 every 8 connection(s)
I20260927 02:45:04.392498 15823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:45:04.395478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 15823
I20260927 02:45:04.395581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:45:04.395808 392 external_mini_cluster.cc:1428] Setting key dcc99946e9b6bf0134ae2d8a5211f7c5
Sep 27 02:45:04 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:04.402326 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.393277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60443 (local address 127.0.98.62:38417)
0927 02:45:04.393325 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.393327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.393544 (+ 217us) server_negotiation.cc:407] Connection header received
0927 02:45:04.394100 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.394101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.394114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.394179 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:04.395236 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.395828 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.397096 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.397303 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.397365 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.399570 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:04.399572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:04.399572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:04.399580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:04.400977 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.401375 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.401377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.401377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.401436 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.401672 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.401674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.401674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.401791 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:04.401854 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.402134 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.402228 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:45:04.402501 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.393255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:60443)
0927 02:45:04.393428 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:04.393465 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:04.394000 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:04.394261 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:04.394265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:04.394853 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:04.395058 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.395064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.395962 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.395963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.396987 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.396990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.397384 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.397385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.397508 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.398023 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:04.398040 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:04.399474 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:04.401067 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:04.401075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:04.401077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:04.401305 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:04.401495 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:04.401496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:04.401496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:04.401612 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:04.401851 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:04.401858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:04.402001 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 02:45:04.402104 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 02:45:04.402279 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:45:04.402976 15952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260927 02:45:04.403075 15952 heartbeater.cc:464] Registering TS with master...
I20260927 02:45:04.403239 15952 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260927 02:45:04.403782 15761 ts_manager.cc:194] Registered new tserver with Master: 93fedfbe449346cd89d1a12daf219cb5 (127.0.98.1:43911)
I20260927 02:45:04.404428 15761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:45:04.419668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:45:04.501097 15957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:04.501329 15957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:04.501354 15957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:04.502540 15957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:04.502599 15957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:04.502626 15957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:45:04.504339 15957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:04.504643 15957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:04.505053 15957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:04.510156 15963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:04.510223 15962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:04.510419 15957 server_base.cc:1034] running on GCE node
W20260927 02:45:04.510442 15965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:04.510663 15957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:04.510906 15957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:04.512053 15957 hybrid_clock.cc:648] HybridClock initialized: now 1790477104512037 us; error 29 us; skew 500 ppm
Sep 27 02:45:04 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:04.514616 15957 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:45:04.515026 15957 webserver.cc:460] Webserver started at http://127.0.98.2:41681/ using document root <none> and password file <none>
I20260927 02:45:04.515206 15957 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:04.515246 15957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:04.515337 15957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:45:04.516323 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "8dcaed5d633f41ac85e8d98d564ebb11"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "528ecab025cfbf3225adf5b1e673703f"
server_key_iv: "143c5da8f0603ab842ef8aeb753cd4bf"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.516664 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8dcaed5d633f41ac85e8d98d564ebb11"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "528ecab025cfbf3225adf5b1e673703f"
server_key_iv: "143c5da8f0603ab842ef8aeb753cd4bf"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.517838 15957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:04.518546 15972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.518709 15957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:45:04.518826 15957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "8dcaed5d633f41ac85e8d98d564ebb11"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "528ecab025cfbf3225adf5b1e673703f"
server_key_iv: "143c5da8f0603ab842ef8aeb753cd4bf"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.518903 15957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:04.533396 15957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:04.535465 15957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:04.535580 15957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:04.535777 15957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:04.536027 15957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:45:04.536062 15957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.536082 15957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:45:04.536098 15957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.541514 15957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44707
I20260927 02:45:04.541627 16085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44707 every 8 connection(s)
I20260927 02:45:04.541888 15957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:45:04.544325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 15957
I20260927 02:45:04.544452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:45:04.544652 392 external_mini_cluster.cc:1428] Setting key 78a4e09a0fe595180f87df9bcc595a15
Sep 27 02:45:04 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:04.551270 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.542600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41381 (local address 127.0.98.62:38417)
0927 02:45:04.542657 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.542659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.542849 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:45:04.543381 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.543382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.543395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.543441 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:04.544506 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.545167 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.546172 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.546363 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.546419 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.548431 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:04.548433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:04.548433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:04.548441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:04.549915 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.550344 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.550345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.550345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.550409 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.550627 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.550628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.550628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.550765 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:04.550837 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.551090 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.551171 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:45:04.551514 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.542584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:41381)
0927 02:45:04.542732 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:04.542768 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:04.543282 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:04.543523 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:04.543528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:04.544078 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:04.544357 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.544364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.545258 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.545259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.546086 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.546089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.546433 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.546434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.546559 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.547022 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:04.547031 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:04.548320 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:04.550000 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:04.550006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:04.550009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:04.550275 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:04.550476 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:04.550477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:04.550477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:04.550569 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:04.550838 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:04.550843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:04.550965 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 02:45:04.551049 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:45:04.551173 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260927 02:45:04.551991 16086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260927 02:45:04.552076 16086 heartbeater.cc:464] Registering TS with master...
I20260927 02:45:04.552295 16086 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:45:04.552718 15761 ts_manager.cc:194] Registered new tserver with Master: 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707)
I20260927 02:45:04.553081 15761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:45:04.573333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:45:04.652894 16091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:04.653123 16091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:04.653156 16091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:04.654351 16091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:04.654399 16091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:04.654435 16091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:45:04.655903 16091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:04.656141 16091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:04.656368 16091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:04.661429 16099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:04.661494 16091 server_base.cc:1034] running on GCE node
W20260927 02:45:04.661425 16097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:04.661425 16096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:04.661866 16091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:04.662098 16091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:04.663249 16091 hybrid_clock.cc:648] HybridClock initialized: now 1790477104663243 us; error 46 us; skew 500 ppm
Sep 27 02:45:04 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:04.665874 16091 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:45:04.666316 16091 webserver.cc:460] Webserver started at http://127.0.98.3:44817/ using document root <none> and password file <none>
I20260927 02:45:04.666538 16091 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:04.666585 16091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:04.666746 16091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:45:04.668023 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "3dcb9e13e3bc43d78677e866501b0c8a"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "8297c1c40ee6607d04a6fb0335fc8f98"
server_key_iv: "62a058faec4fbb834b61c7f5e1eb29ed"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.668354 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3dcb9e13e3bc43d78677e866501b0c8a"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "8297c1c40ee6607d04a6fb0335fc8f98"
server_key_iv: "62a058faec4fbb834b61c7f5e1eb29ed"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.669693 16091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:45:04.670531 16106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.670691 16091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:45:04.670796 16091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "3dcb9e13e3bc43d78677e866501b0c8a"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "8297c1c40ee6607d04a6fb0335fc8f98"
server_key_iv: "62a058faec4fbb834b61c7f5e1eb29ed"
server_key_version: "encryptionkey@0"
I20260927 02:45:04.670861 16091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:04.695693 16091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:04.697863 16091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:04.698007 16091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:04.698221 16091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:04.698508 16091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:45:04.698540 16091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.698570 16091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:45:04.698594 16091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:04.703999 16091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37827
I20260927 02:45:04.704067 16219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37827 every 8 connection(s)
I20260927 02:45:04.704355 16091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:45:04.708247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 16091
I20260927 02:45:04.708415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:45:04.708609 392 external_mini_cluster.cc:1428] Setting key a8bdebee24cc4a572e8cd1291fd6a5b2
Sep 27 02:45:04 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:04.713507 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.705058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44055 (local address 127.0.98.62:38417)
0927 02:45:04.705125 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.705127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.705311 (+ 184us) server_negotiation.cc:407] Connection header received
0927 02:45:04.705840 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.705841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.705851 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.705908 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:04.706891 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.707451 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.708651 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.708915 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.708986 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.711135 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:04.711136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:04.711136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:04.711143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:04.712304 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.712686 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.712687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.712687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.712755 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.712951 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.712951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.712952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.713045 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:04.713089 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.713372 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.713442 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:45:04.713596 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.705012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:44055)
0927 02:45:04.705176 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:04.705238 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:04.705720 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:04.705992 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:04.705996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:04.706597 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:04.706805 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.706810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.707554 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.707555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.708571 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.708574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.709020 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.709021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.709165 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.709738 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:04.709746 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:04.711014 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:04.712371 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:04.712376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:04.712378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:04.712622 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:04.712809 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:04.712809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:04.712810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:04.712906 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:04.713089 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:04.713093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:04.713206 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 02:45:04.713308 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:45:04.713425 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260927 02:45:04.714017 16220 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260927 02:45:04.714119 16220 heartbeater.cc:464] Registering TS with master...
I20260927 02:45:04.714296 16220 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260927 02:45:04.714634 15761 ts_manager.cc:194] Registered new tserver with Master: 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827)
I20260927 02:45:04.715019 15761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44055
I20260927 02:45:04.720346 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:45:04.727885 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.721739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.0.98.62:38417)
0927 02:45:04.721785 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.721787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.721912 (+ 125us) server_negotiation.cc:407] Connection header received
0927 02:45:04.721997 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.721998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.722008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.722067 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:04.722600 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.723182 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.723960 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.724151 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.724209 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.725553 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:04.725554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:04.725555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:04.725562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:04.726893 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.727290 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.727291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.727292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.727343 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:04.727486 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:04.727486 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:04.727486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:04.727617 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:04.727682 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.727761 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.727820 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:45:04.729539 15761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:45:04.731083 15761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:45:04.737488 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.733131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37827 (local address 127.0.0.1:41642)
0927 02:45:04.733708 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:04.733716 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:04.733764 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:04.734062 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:04.734063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:04.734065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:04.734100 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.734101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.735363 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.735364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.736241 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.736243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.737176 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.737176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.737249 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.737265 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:45:04.737342 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:45:04.737398 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":67,"threads_started":1}
I20260927 02:45:04.737502 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.733375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.0.98.3:37827)
0927 02:45:04.733618 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.733621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.733767 (+ 146us) server_negotiation.cc:407] Connection header received
0927 02:45:04.733824 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.733887 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.733909 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.733980 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:04.734168 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.735282 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.736313 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.737069 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.737145 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.737215 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.737376 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.737417 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260927 02:45:04.737705 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.733381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.0.98.1:43911)
0927 02:45:04.733555 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.733560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.733891 (+ 331us) server_negotiation.cc:407] Connection header received
0927 02:45:04.734046 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.734049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.734063 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.734112 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:04.734316 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.734990 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.736519 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.737205 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.737268 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.737311 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.737545 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.737617 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260927 02:45:04.737835 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.733178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43911 (local address 127.0.0.1:45868)
0927 02:45:04.733831 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:04.733839 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:04.733943 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:04.734212 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:04.734212 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:04.734214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:04.734254 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.734256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.735079 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.735080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.736445 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.736447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.737288 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.737289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.737349 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.737352 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:45:04.737411 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:45:04.737445 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":42,"thread_start_us":47,"threads_started":1}
I20260927 02:45:04.738382 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.733135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44707 (local address 127.0.0.1:41824)
0927 02:45:04.733482 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:04.733521 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:04.733592 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:04.733809 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:04.733812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:04.733815 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:04.733888 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.733890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.735194 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.735195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.736444 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.736447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.738150 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.738151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.738221 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.738225 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:45:04.738273 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:45:04.738316 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":3,"thread_start_us":65,"threads_started":1}
I20260927 02:45:04.738785 16154 tablet_service.cc:1467] Processing CreateTablet for tablet b1fe66cb4723446389cf57b236635d7c (DEFAULT_TABLE table=test-table [id=659911f58b124ea3a8cff518bb358b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:45:04.738817 15886 tablet_service.cc:1467] Processing CreateTablet for tablet b1fe66cb4723446389cf57b236635d7c (DEFAULT_TABLE table=test-table [id=659911f58b124ea3a8cff518bb358b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:45:04.739081 15886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fe66cb4723446389cf57b236635d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:04.739100 16154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fe66cb4723446389cf57b236635d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:04.741832 16234 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: Bootstrap starting.
I20260927 02:45:04.741846 16235 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Bootstrap starting.
I20260927 02:45:04.742206 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.733327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.0.98.2:44707)
0927 02:45:04.733472 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.733477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.733582 (+ 105us) server_negotiation.cc:407] Connection header received
0927 02:45:04.733693 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.733696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.733711 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.733789 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:04.733982 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.735088 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.737346 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.737996 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.738060 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.738116 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.742056 (+ 3940us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.742117 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260927 02:45:04.742471 16235 tablet_bootstrap.cc:654] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:04.742544 16234 tablet_bootstrap.cc:654] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:04.742995 16234 log.cc:824] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:04.743021 16235 log.cc:824] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:04.743060 16020 tablet_service.cc:1467] Processing CreateTablet for tablet b1fe66cb4723446389cf57b236635d7c (DEFAULT_TABLE table=test-table [id=659911f58b124ea3a8cff518bb358b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:45:04.743382 16020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fe66cb4723446389cf57b236635d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:04.743671 16234 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: No bootstrap required, opened a new log
I20260927 02:45:04.743726 16234 ts_tablet_manager.cc:1397] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:45:04.745011 16235 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: No bootstrap required, opened a new log
I20260927 02:45:04.745090 16235 ts_tablet_manager.cc:1397] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:45:04.745364 16234 raft_consensus.cc:348] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:04.745541 16234 raft_consensus.cc:374] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:04.745573 16234 raft_consensus.cc:729] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dcb9e13e3bc43d78677e866501b0c8a, State: Initialized, Role: FOLLOWER
I20260927 02:45:04.745720 16234 consensus_queue.cc:260] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:04.745990 16234 ts_tablet_manager.cc:1428] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:45:04.745982 16238 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Bootstrap starting.
I20260927 02:45:04.746491 16238 tablet_bootstrap.cc:654] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:04.746788 16235 raft_consensus.cc:348] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:04.746920 16235 raft_consensus.cc:374] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:04.746941 16235 raft_consensus.cc:729] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93fedfbe449346cd89d1a12daf219cb5, State: Initialized, Role: FOLLOWER
I20260927 02:45:04.746935 16238 log.cc:824] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:04.747048 16235 consensus_queue.cc:260] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:04.747288 16235 ts_tablet_manager.cc:1428] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:45:04.747324 15952 heartbeater.cc:503] Master 127.0.98.62:38417 was elected leader, sending a full tablet report...
I20260927 02:45:04.747534 16220 heartbeater.cc:503] Master 127.0.98.62:38417 was elected leader, sending a full tablet report...
I20260927 02:45:04.747695 16238 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: No bootstrap required, opened a new log
I20260927 02:45:04.747792 16238 ts_tablet_manager.cc:1397] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:45:04.749366 16238 raft_consensus.cc:348] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:04.749498 16238 raft_consensus.cc:374] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:04.749518 16238 raft_consensus.cc:729] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dcaed5d633f41ac85e8d98d564ebb11, State: Initialized, Role: FOLLOWER
I20260927 02:45:04.749607 16238 consensus_queue.cc:260] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:04.749861 16238 ts_tablet_manager.cc:1428] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:45:04.749951 16086 heartbeater.cc:503] Master 127.0.98.62:38417 was elected leader, sending a full tablet report...
W20260927 02:45:04.792512 16087 tablet.cc:2369] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:45:04.893266 15953 tablet.cc:2369] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:45:04.955015 16221 tablet.cc:2369] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:45:04.964566 16239 raft_consensus.cc:482] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:45:04.964696 16239 raft_consensus.cc:504] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:04.965087 16239 leader_election.cc:288] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707), 93fedfbe449346cd89d1a12daf219cb5 (127.0.98.1:43911)
I20260927 02:45:04.968103 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.965312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43911 (local address 127.0.98.3:44931)
0927 02:45:04.965367 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:04.965379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:04.965427 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:04.965584 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:04.965585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:04.965587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:04.965651 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.965652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.966381 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.966383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.967306 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.967310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.967847 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.967848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.967919 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.967923 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:45:04.967974 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:45:04.968018 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 02:45:04.968112 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.965365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44931 (local address 127.0.98.1:43911)
0927 02:45:04.965416 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.965418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.965429 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:45:04.965495 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.965496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.965507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.965569 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:04.965733 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.966287 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.967394 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.967777 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.967840 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.967852 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.967992 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.968041 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:45:04.968133 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.965364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60323 (local address 127.0.98.2:44707)
0927 02:45:04.965406 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:04.965408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:04.965588 (+ 180us) server_negotiation.cc:407] Connection header received
0927 02:45:04.965618 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:04.965619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:04.965632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:04.965679 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:04.965819 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.966301 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.967435 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.967809 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.967873 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.967883 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:04.968043 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 02:45:04.968080 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:45:04.968317 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:04.965317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44707 (local address 127.0.98.3:60323)
0927 02:45:04.965494 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:04.965502 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:04.965551 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:04.965689 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:04.965690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:04.965691 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:04.965748 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.965750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.966380 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.966381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.967320 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:04.967323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:04.967892 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:04.967894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:04.967957 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:04.967961 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:45:04.968198 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 02:45:04.968245 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":73,"thread_start_us":96,"threads_started":1}
I20260927 02:45:04.968441 15906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fe66cb4723446389cf57b236635d7c" candidate_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93fedfbe449346cd89d1a12daf219cb5" is_pre_election: true
I20260927 02:45:04.968616 15906 raft_consensus.cc:2454] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dcb9e13e3bc43d78677e866501b0c8a in term 0.
I20260927 02:45:04.968639 16040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fe66cb4723446389cf57b236635d7c" candidate_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" is_pre_election: true
I20260927 02:45:04.968781 16040 raft_consensus.cc:2454] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dcb9e13e3bc43d78677e866501b0c8a in term 0.
I20260927 02:45:04.968838 16108 leader_election.cc:302] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dcb9e13e3bc43d78677e866501b0c8a, 93fedfbe449346cd89d1a12daf219cb5; no voters:
I20260927 02:45:04.968959 16239 raft_consensus.cc:2781] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:45:04.969019 16239 raft_consensus.cc:482] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:45:04.969039 16239 raft_consensus.cc:3037] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:04.969762 16239 raft_consensus.cc:504] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:04.969879 16239 leader_election.cc:288] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [CANDIDATE]: Term 1 election: Requested vote from peers 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707), 93fedfbe449346cd89d1a12daf219cb5 (127.0.98.1:43911)
I20260927 02:45:04.970070 16040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fe66cb4723446389cf57b236635d7c" candidate_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dcaed5d633f41ac85e8d98d564ebb11"
I20260927 02:45:04.970098 15906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fe66cb4723446389cf57b236635d7c" candidate_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93fedfbe449346cd89d1a12daf219cb5"
I20260927 02:45:04.970153 16040 raft_consensus.cc:3037] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:04.970184 15906 raft_consensus.cc:3037] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:04.970928 15906 raft_consensus.cc:2454] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dcb9e13e3bc43d78677e866501b0c8a in term 1.
I20260927 02:45:04.970986 16040 raft_consensus.cc:2454] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dcb9e13e3bc43d78677e866501b0c8a in term 1.
I20260927 02:45:04.971095 16108 leader_election.cc:302] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dcb9e13e3bc43d78677e866501b0c8a, 93fedfbe449346cd89d1a12daf219cb5; no voters:
I20260927 02:45:04.971208 16239 raft_consensus.cc:2781] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:45:04.971375 16239 raft_consensus.cc:686] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 LEADER]: Becoming Leader. State: Replica: 3dcb9e13e3bc43d78677e866501b0c8a, State: Running, Role: LEADER
I20260927 02:45:04.971462 16239 consensus_queue.cc:237] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:04.972316 15761 catalog_manager.cc:5495] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } health_report { overall_health: UNKNOWN } } }
I20260927 02:45:05.028553 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:05.025921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.98.3:37827)
0927 02:45:05.025972 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:05.025974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:05.025991 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:45:05.026116 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:05.026117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:05.026143 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:05.026209 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:45:05.026369 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:05.026849 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:05.027727 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:05.027917 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:05.027984 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:05.028214 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:45:05.028326 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:45:05.028426 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:05.028447 (+ 21us) server_negotiation.cc:299] Negotiation successful
0927 02:45:05.028473 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:45:05.030483 16040 raft_consensus.cc:1261] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Refusing update from remote peer 3dcb9e13e3bc43d78677e866501b0c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:45:05.030483 15906 raft_consensus.cc:1261] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Refusing update from remote peer 3dcb9e13e3bc43d78677e866501b0c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:45:05.030849 16244 consensus_queue.cc:1035] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:45:05.031006 16239 consensus_queue.cc:1035] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:45:05.033581 16247 mvcc.cc:205] Tried to move back new op lower bound from 7333794222202253312 to 7333794221963702272. Current Snapshot: MvccSnapshot[applied={T|T < 7333794222202253312}]
I20260927 02:45:05.034225 16249 mvcc.cc:205] Tried to move back new op lower bound from 7333794222202253312 to 7333794221963702272. Current Snapshot: MvccSnapshot[applied={T|T < 7333794222202253312}]
I20260927 02:45:05.036041 15761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:45:05.036139 15761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:45:05.036847 15761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Sending DeleteTablet for 3 replicas of tablet b1fe66cb4723446389cf57b236635d7c
I20260927 02:45:05.037147 15886 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fe66cb4723446389cf57b236635d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45868
I20260927 02:45:05.037141 16020 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fe66cb4723446389cf57b236635d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41824
I20260927 02:45:05.037303 16154 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fe66cb4723446389cf57b236635d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41642
I20260927 02:45:05.037497 16256 tablet_replica.cc:330] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: stopping tablet replica
I20260927 02:45:05.037528 16258 tablet_replica.cc:330] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: stopping tablet replica
I20260927 02:45:05.037544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 15823
I20260927 02:45:05.037672 16258 raft_consensus.cc:2229] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:45:05.037798 16258 raft_consensus.cc:2258] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:45:05.038864 16258 ts_tablet_manager.cc:1905] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:45:05.040516 16257 tablet_replica.cc:330] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: stopping tablet replica
I20260927 02:45:05.040618 16258 ts_tablet_manager.cc:1918] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:45:05.040711 16257 raft_consensus.cc:2229] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:45:05.040724 16258 log.cc:1196] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/b1fe66cb4723446389cf57b236635d7c
I20260927 02:45:05.040875 16257 raft_consensus.cc:2258] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:45:05.040972 16258 ts_tablet_manager.cc:1939] T b1fe66cb4723446389cf57b236635d7c P 3dcb9e13e3bc43d78677e866501b0c8a: Deleting consensus metadata
I20260927 02:45:05.041476 15745 catalog_manager.cc:4841] TS 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827): tablet b1fe66cb4723446389cf57b236635d7c (table test-table [id=659911f58b124ea3a8cff518bb358b76]) successfully deleted
W20260927 02:45:05.042013 15746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43911 (error 108)
I20260927 02:45:05.042254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 15957
I20260927 02:45:05.042876 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:05.042364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43911 (local address 127.0.0.1:45874)
0927 02:45:05.042810 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:05.042848 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426}
W20260927 02:45:05.042982 15746 catalog_manager.cc:4584] TS 93fedfbe449346cd89d1a12daf219cb5 (127.0.98.1:43911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1fe66cb4723446389cf57b236635d7c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43911: connect: Connection refused (error 111)
I20260927 02:45:05.046627 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:05.046506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44707 (local address 127.0.0.1:41836)
0927 02:45:05.046557 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:05.046593 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:45:05.046713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 16091
I20260927 02:45:05.050625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 15729
I20260927 02:45:25.054960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--webserver_interface=127.0.98.62
--webserver_port=34531
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:45:25.136488 16260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:25.136735 16260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:25.136763 16260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:25.137998 16260 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:45:25.138046 16260 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:45:25.138059 16260 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:45:25.138069 16260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:45:25.138080 16260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:25.138092 16260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:25.138103 16260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:45:25.138113 16260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:45:25.139717 16260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:25.139947 16260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:25.140134 16260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:25.148509 16269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:25.148640 16267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:25.148710 16260 server_base.cc:1034] running on GCE node
W20260927 02:45:25.148535 16266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:25.149072 16260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:25.149385 16260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:25.150533 16260 hybrid_clock.cc:648] HybridClock initialized: now 1790477125150514 us; error 37 us; skew 500 ppm
Sep 27 02:45:25 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:25.157481 16260 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:45:25.158015 16260 webserver.cc:460] Webserver started at http://127.0.98.62:34531/ using document root <none> and password file <none>
I20260927 02:45:25.158304 16260 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:25.158366 16260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:25.159895 16260 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:45:25.160769 16276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:25.160943 16260 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:25.161034 16260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "54557d141cc74e43bab5bb488bb58ced"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "e813bba08ed1d2fc2f9af81045e0ea92"
server_key_iv: "3a369018bc3b1384f55a13a8f4c128f7"
server_key_version: "encryptionkey@0"
I20260927 02:45:25.161438 16260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:25.172416 16260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:25.174543 16260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:25.174688 16260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:25.178403 16260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38417
I20260927 02:45:25.178437 16328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38417 every 8 connection(s)
I20260927 02:45:25.178797 16260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:45:25.179792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 16260
I20260927 02:45:25.180032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43911
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38417
--webserver_port=43753
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:45:25.180320 16329 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:45:25.180979 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Bootstrap starting.
I20260927 02:45:25.189296 16329 log.cc:824] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:25.191298 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:45:25.191557 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Bootstrap complete.
I20260927 02:45:25.193849 16329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:25.194085 16329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54557d141cc74e43bab5bb488bb58ced, State: Initialized, Role: FOLLOWER
I20260927 02:45:25.194238 16329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:25.194308 16329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:45:25.194344 16329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:45:25.194393 16329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:45:25.195469 16329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:25.195564 16329 leader_election.cc:302] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54557d141cc74e43bab5bb488bb58ced; no voters:
I20260927 02:45:25.195719 16329 leader_election.cc:288] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:45:25.195763 16333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:45:25.195931 16333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 2 LEADER]: Becoming Leader. State: Replica: 54557d141cc74e43bab5bb488bb58ced, State: Running, Role: LEADER
I20260927 02:45:25.195994 16329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:45:25.196048 16333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:25.196362 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "54557d141cc74e43bab5bb488bb58ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260927 02:45:25.196466 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: This master's current role is: LEADER
I20260927 02:45:25.196830 16335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54557d141cc74e43bab5bb488bb58ced. Latest consensus state: current_term: 2 leader_uuid: "54557d141cc74e43bab5bb488bb58ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260927 02:45:25.196899 16335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: This master's current role is: LEADER
I20260927 02:45:25.197033 16339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:45:25.197885 16339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:45:25.198127 16339 catalog_manager.cc:1259] Loaded cluster ID: 2df93773bcaa404090f78fb718754a8e
I20260927 02:45:25.198258 16339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:45:25.199394 16339 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:45:25.199572 16339 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Loaded TSK: 0
I20260927 02:45:25.210399 16339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Generated new TSK 1
I20260927 02:45:25.210523 16339 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:45:25.263055 16331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:25.263274 16331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:25.263305 16331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:25.264555 16331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:25.264597 16331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:25.264633 16331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:45:25.266119 16331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43753
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:25.266348 16331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:25.266544 16331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:25.272359 16357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:25.272493 16359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:25.272627 16331 server_base.cc:1034] running on GCE node
W20260927 02:45:25.272374 16356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:25.272958 16331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:25.273257 16331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:25.274443 16331 hybrid_clock.cc:648] HybridClock initialized: now 1790477125274420 us; error 41 us; skew 500 ppm
Sep 27 02:45:25 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:25.277585 16331 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:45:25.278219 16331 webserver.cc:460] Webserver started at http://127.0.98.1:43753/ using document root <none> and password file <none>
I20260927 02:45:25.278474 16331 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:25.278559 16331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:25.280514 16331 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:45:25.281402 16366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:25.281615 16331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:25.281716 16331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "93fedfbe449346cd89d1a12daf219cb5"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "f6e3b36cc39c952b1e8407a0783bddef"
server_key_iv: "896628e08b3bd800971ea86e7260afb4"
server_key_version: "encryptionkey@0"
I20260927 02:45:25.282173 16331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:25.325243 16331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:25.327927 16331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:25.328075 16331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:25.328334 16331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:25.328890 16373 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:45:25.330415 16331 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:45:25.330476 16331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:45:25.330507 16331 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:45:25.331362 16331 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:45:25.331404 16331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:45:25.331503 16373 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Bootstrap starting.
I20260927 02:45:25.338421 16331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43911
I20260927 02:45:25.338701 16480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43911 every 8 connection(s)
I20260927 02:45:25.338866 16331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:45:25.342224 16373 log.cc:824] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:25.345160 16373 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:45:25.345429 16373 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Bootstrap complete.
I20260927 02:45:25.345789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 16331
I20260927 02:45:25.346060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44707
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38417
--webserver_port=41681
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:45:25.346446 16373 ts_tablet_manager.cc:1397] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260927 02:45:25.347743 16373 raft_consensus.cc:348] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.347983 16373 raft_consensus.cc:729] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93fedfbe449346cd89d1a12daf219cb5, State: Initialized, Role: FOLLOWER
Sep 27 02:45:25 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:25.348711 16373 consensus_queue.cc:260] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.349020 16373 ts_tablet_manager.cc:1428] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260927 02:45:25.352174 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.339581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:47127)
0927 02:45:25.339757 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:25.339878 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:25.340628 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:25.342883 (+ 2255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:25.342888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:25.343214 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:25.343469 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.343475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.345099 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.345100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.346115 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.346118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.346861 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.346862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.346965 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.347487 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:25.347495 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:25.349018 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:25.350676 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:25.350682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:25.350685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:25.350898 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:25.351123 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:25.351124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:25.351125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:25.351241 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:25.351576 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:25.351582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:25.351714 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 02:45:25.351804 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:45:25.351978 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260927 02:45:25.352173 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.339688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47127 (local address 127.0.98.62:38417)
0927 02:45:25.342474 (+ 2786us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.342484 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.342517 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:45:25.342646 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.342652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.342697 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.342809 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:25.343607 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.344970 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.346228 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.346679 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.346748 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.349118 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:25.349122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:25.349122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:25.349133 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:25.350599 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.350972 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.350975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.350975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.351065 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.351297 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.351297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.351298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.351515 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:25.351584 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.351866 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.351987 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2723,"thread_start_us":84,"threads_started":1}
I20260927 02:45:25.352880 16481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260927 02:45:25.352989 16481 heartbeater.cc:464] Registering TS with master...
I20260927 02:45:25.353184 16481 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260927 02:45:25.353891 16293 ts_manager.cc:194] Registered new tserver with Master: 93fedfbe449346cd89d1a12daf219cb5 (127.0.98.1:43911)
I20260927 02:45:25.354967 16293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47127
I20260927 02:45:25.356011 16481 heartbeater.cc:503] Master 127.0.98.62:38417 was elected leader, sending a full tablet report...
Sep 27 02:45:25 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:45:25.362901 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.354527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.0.98.1:43911)
0927 02:45:25.354709 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.354713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.354728 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:45:25.354840 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.354842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.354858 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.354941 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:25.355526 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.356375 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.357454 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.357728 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.357780 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.359856 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:25.359862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:25.359862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:25.359869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:25.361590 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.362056 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.362059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.362059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.362127 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.362360 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.362361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.362361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.362493 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:25.362559 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.362756 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.362836 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260927 02:45:25.362948 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.354443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43911 (local address 127.0.0.1:59056)
0927 02:45:25.354650 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:25.354680 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:25.354744 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:25.354947 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:25.354950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:25.355398 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:25.355457 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.355459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.356452 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.356452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.357379 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.357382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.357789 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.357792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.357867 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.358303 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:25.358309 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:25.359757 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:25.361669 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:25.361677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:25.361679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:25.361948 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:25.362236 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:25.362237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:25.362237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:25.362305 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:25.362560 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:25.362568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:25.362629 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 02:45:25.362718 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:45:25.362846 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
I20260927 02:45:25.363631 16413 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fe66cb4723446389cf57b236635d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59056
I20260927 02:45:25.364241 16490 tablet_replica.cc:330] stopping tablet replica
I20260927 02:45:25.364336 16490 raft_consensus.cc:2229] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:45:25.364436 16490 raft_consensus.cc:2258] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:45:25.364761 16490 ts_tablet_manager.cc:1905] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:45:25.367094 16490 ts_tablet_manager.cc:1918] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:45:25.367158 16490 log.cc:1196] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal/wals/b1fe66cb4723446389cf57b236635d7c
I20260927 02:45:25.367389 16490 ts_tablet_manager.cc:1939] T b1fe66cb4723446389cf57b236635d7c P 93fedfbe449346cd89d1a12daf219cb5: Deleting consensus metadata
I20260927 02:45:25.367851 16278 catalog_manager.cc:4841] TS 93fedfbe449346cd89d1a12daf219cb5 (127.0.98.1:43911): tablet b1fe66cb4723446389cf57b236635d7c (table test-table [id=659911f58b124ea3a8cff518bb358b76]) successfully deleted
W20260927 02:45:25.430317 16486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:25.430553 16486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:25.430588 16486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:25.431789 16486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:25.431849 16486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:25.431883 16486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:45:25.433446 16486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41681
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:25.433662 16486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:25.433853 16486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:25.439253 16496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:25.439277 16486 server_base.cc:1034] running on GCE node
W20260927 02:45:25.439277 16495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:25.439461 16498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:25.439715 16486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:25.439975 16486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:25.441148 16486 hybrid_clock.cc:648] HybridClock initialized: now 1790477125441128 us; error 38 us; skew 500 ppm
Sep 27 02:45:25 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:25.443799 16486 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:45:25.444293 16486 webserver.cc:460] Webserver started at http://127.0.98.2:41681/ using document root <none> and password file <none>
I20260927 02:45:25.444543 16486 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:25.444609 16486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:25.446058 16486 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:25.446805 16505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:25.446983 16486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:25.447108 16486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "8dcaed5d633f41ac85e8d98d564ebb11"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "528ecab025cfbf3225adf5b1e673703f"
server_key_iv: "143c5da8f0603ab842ef8aeb753cd4bf"
server_key_version: "encryptionkey@0"
I20260927 02:45:25.447443 16486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:25.459635 16486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:25.461671 16486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:25.461795 16486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:25.461997 16486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:25.462419 16512 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:45:25.463896 16486 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:45:25.463945 16486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:45:25.463977 16486 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:45:25.464685 16486 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:45:25.464717 16486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:25.464794 16512 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Bootstrap starting.
I20260927 02:45:25.470281 16486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44707
I20260927 02:45:25.470453 16619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44707 every 8 connection(s)
I20260927 02:45:25.470667 16486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:45:25.472026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 16486
I20260927 02:45:25.472407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37827
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38417
--webserver_port=44817
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:45:25.474923 16512 log.cc:824] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:45:25 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:25.480525 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.471476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36293 (local address 127.0.98.62:38417)
0927 02:45:25.471534 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.471535 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.471709 (+ 174us) server_negotiation.cc:407] Connection header received
0927 02:45:25.472219 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.472221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.472239 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.472289 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:25.473031 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.473529 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.474943 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.475200 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.475256 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.477653 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:25.477655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:25.477655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:25.477662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:25.479008 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.479487 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.479488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.479488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.479538 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.479808 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.479809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.479809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.479914 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:25.479960 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.480312 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.480432 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:45:25.480991 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.471408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:36293)
0927 02:45:25.471576 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:25.471618 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:25.472083 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:25.472350 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:25.472354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:25.472730 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:25.472933 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.472939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.473632 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.473633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.474867 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.474870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.475290 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.475291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.475418 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.475980 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:25.475996 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:25.477546 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:25.479092 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:25.479100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:25.479103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:25.479413 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:25.479599 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:25.479600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:25.479601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:25.479742 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:25.479973 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:25.479979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:25.480170 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 02:45:25.480291 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 02:45:25.480500 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260927 02:45:25.481515 16620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260927 02:45:25.481616 16620 heartbeater.cc:464] Registering TS with master...
I20260927 02:45:25.481801 16620 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260927 02:45:25.482486 16293 ts_manager.cc:194] Registered new tserver with Master: 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707)
I20260927 02:45:25.482620 16512 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:45:25.483073 16512 tablet_bootstrap.cc:492] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Bootstrap complete.
I20260927 02:45:25.483111 16293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36293
I20260927 02:45:25.484057 16512 ts_tablet_manager.cc:1397] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.013s
I20260927 02:45:25.485256 16512 raft_consensus.cc:348] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.485562 16512 raft_consensus.cc:729] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dcaed5d633f41ac85e8d98d564ebb11, State: Initialized, Role: FOLLOWER
I20260927 02:45:25.485689 16512 consensus_queue.cc:260] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.485939 16512 ts_tablet_manager.cc:1428] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:45:25.485956 16620 heartbeater.cc:503] Master 127.0.98.62:38417 was elected leader, sending a full tablet report...
I20260927 02:45:25.487898 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.482923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.0.98.2:44707)
0927 02:45:25.484507 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.484511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.484522 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:45:25.484554 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.484556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.484573 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.484815 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:25.484853 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.485808 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.487059 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.487546 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.487606 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.487663 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.487796 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.487830 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1547,"thread_start_us":81,"threads_started":1}
I20260927 02:45:25.487897 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.482804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44707 (local address 127.0.0.1:53940)
0927 02:45:25.482844 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:25.482855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:25.482895 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:25.484696 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:25.484697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:25.484699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:25.484752 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.484754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.485907 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.485909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.486986 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.486988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.487620 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.487621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.487698 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.487725 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 02:45:25.487774 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:45:25.487825 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 02:45:25.488299 16554 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fe66cb4723446389cf57b236635d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53940
I20260927 02:45:25.488885 16627 tablet_replica.cc:330] stopping tablet replica
I20260927 02:45:25.488988 16627 raft_consensus.cc:2229] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:45:25.489110 16627 raft_consensus.cc:2258] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:45:25.489375 16627 ts_tablet_manager.cc:1905] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:45:25.491475 16627 ts_tablet_manager.cc:1918] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:45:25.491545 16627 log.cc:1196] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/b1fe66cb4723446389cf57b236635d7c
I20260927 02:45:25.491765 16627 ts_tablet_manager.cc:1939] T b1fe66cb4723446389cf57b236635d7c P 8dcaed5d633f41ac85e8d98d564ebb11: Deleting consensus metadata
I20260927 02:45:25.492228 16279 catalog_manager.cc:4841] TS 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707): tablet b1fe66cb4723446389cf57b236635d7c (table test-table [id=659911f58b124ea3a8cff518bb358b76]) successfully deleted
W20260927 02:45:25.555490 16623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:25.555737 16623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:25.555766 16623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:25.557075 16623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:25.557123 16623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:25.557152 16623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:45:25.558699 16623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44817
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:25.558938 16623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:25.559154 16623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:25.564133 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:25.564092 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:25.564108 16635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:25.564283 16623 server_base.cc:1034] running on GCE node
I20260927 02:45:25.564694 16623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:25.564954 16623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:25.566150 16623 hybrid_clock.cc:648] HybridClock initialized: now 1790477125566128 us; error 37 us; skew 500 ppm
Sep 27 02:45:25 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:25.568898 16623 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:45:25.569368 16623 webserver.cc:460] Webserver started at http://127.0.98.3:44817/ using document root <none> and password file <none>
I20260927 02:45:25.569564 16623 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:25.569629 16623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:25.571197 16623 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:45:25.571965 16642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:25.572206 16623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:45:25.572296 16623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "3dcb9e13e3bc43d78677e866501b0c8a"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "8297c1c40ee6607d04a6fb0335fc8f98"
server_key_iv: "62a058faec4fbb834b61c7f5e1eb29ed"
server_key_version: "encryptionkey@0"
I20260927 02:45:25.572695 16623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:25.622725 16623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:25.625263 16623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:25.625456 16623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:25.625716 16623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:25.626080 16623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:45:25.626125 16623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:25.626156 16623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:45:25.626180 16623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:25.633407 16623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37827
I20260927 02:45:25.633453 16755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37827 every 8 connection(s)
I20260927 02:45:25.633891 16623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:45:25.638865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 16623
Sep 27 02:45:25 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:45:25.646452 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.634717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40757 (local address 127.0.98.62:38417)
0927 02:45:25.634799 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.634802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.635095 (+ 293us) server_negotiation.cc:407] Connection header received
0927 02:45:25.635829 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.635830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.635843 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.635898 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:25.637323 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.638097 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.639849 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.640155 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.640219 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.642990 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:25.642992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:25.642992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:25.643001 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:25.644588 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.645325 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.645326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.645326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.645381 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.645695 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.645696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.645696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.645824 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:25.645890 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.646254 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.646342 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:45:25.646589 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.639293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.0.98.62:38417)
0927 02:45:25.639425 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.639427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.639525 (+ 98us) server_negotiation.cc:407] Connection header received
0927 02:45:25.639598 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.639599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.639612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.639678 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:25.640363 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.641013 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.642071 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.642390 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.642461 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.643827 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:25.643829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:25.643829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:25.643836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:25.645244 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.645661 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.645662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.645663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.645717 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.646031 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.646032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.646032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.646130 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:25.646296 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.646331 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.646408 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260927 02:45:25.647037 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.634723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:40757)
0927 02:45:25.634920 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:25.634973 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:25.635685 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:25.636006 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:25.636011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:25.636883 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:25.637220 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.637229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.638191 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.638192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.639758 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.639761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.640245 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.640246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.640434 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.640988 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:25.640998 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:25.642850 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:25.644885 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:25.644894 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:25.644897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:25.645236 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:25.645438 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:25.645439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:25.645439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:25.645598 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:25.645882 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:25.645889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:25.646076 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 02:45:25.646585 (+ 509us) client_negotiation.cc:240] Negotiation successful
0927 02:45:25.646787 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260927 02:45:25.647679 16756 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260927 02:45:25.647782 16756 heartbeater.cc:464] Registering TS with master...
I20260927 02:45:25.647958 16756 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260927 02:45:25.648319 16293 ts_manager.cc:194] Registered new tserver with Master: 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827)
I20260927 02:45:25.648708 16293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40757
I20260927 02:45:25.658800 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:45:25.661741 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.659303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.0.98.62:38417)
0927 02:45:25.659350 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.659352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.659396 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:45:25.659482 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.659483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.659520 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.659579 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:45:25.659681 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.660165 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.661082 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.661285 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.661343 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.661475 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:45:25.661523 (+ 48us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 02:45:25.661650 (+ 127us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50176: token expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 02:45:25.661820 16760 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50176: token expired
W20260927 02:45:25.661875 16225 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:38417)
I20260927 02:45:25.667867 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.662107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.0.98.62:38417)
0927 02:45:25.662158 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.662160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.662229 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:45:25.662248 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.662249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.662256 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.662298 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:25.662743 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.663204 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.663916 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.664094 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.664214 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.665382 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:25.665384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:25.665384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:25.665390 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:25.666751 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.667115 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.667116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.667116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.667173 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:25.667365 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:25.667366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:25.667366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:25.667484 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:25.667581 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.667731 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.667798 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:45:25.669073 16293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:45:25.670663 16293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:45:25.672573 16554 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c (DEFAULT_TABLE table=test-table [id=f4872510787f4babb17ba1e2ba06e141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:45:25.672562 16413 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c (DEFAULT_TABLE table=test-table [id=f4872510787f4babb17ba1e2ba06e141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:45:25.672747 16554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:25.672747 16413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:25.674232 16512 tablet_bootstrap.cc:492] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: Bootstrap starting.
I20260927 02:45:25.674229 16373 tablet_bootstrap.cc:492] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Bootstrap starting.
I20260927 02:45:25.674499 16512 tablet_bootstrap.cc:654] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:25.674551 16373 tablet_bootstrap.cc:654] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:25.674969 16512 tablet_bootstrap.cc:492] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: No bootstrap required, opened a new log
I20260927 02:45:25.675020 16512 ts_tablet_manager.cc:1397] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:25.675127 16512 raft_consensus.cc:348] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.675200 16512 raft_consensus.cc:374] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:25.675248 16512 raft_consensus.cc:729] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dcaed5d633f41ac85e8d98d564ebb11, State: Initialized, Role: FOLLOWER
I20260927 02:45:25.675276 16373 tablet_bootstrap.cc:492] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: No bootstrap required, opened a new log
I20260927 02:45:25.675297 16512 consensus_queue.cc:260] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.675331 16373 ts_tablet_manager.cc:1397] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:45:25.675377 16512 ts_tablet_manager.cc:1428] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:25.675668 16373 raft_consensus.cc:348] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.675736 16373 raft_consensus.cc:374] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:25.675792 16373 raft_consensus.cc:729] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93fedfbe449346cd89d1a12daf219cb5, State: Initialized, Role: FOLLOWER
I20260927 02:45:25.675861 16373 consensus_queue.cc:260] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.675949 16373 ts_tablet_manager.cc:1428] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:45:25.677026 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.672333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37827 (local address 127.0.0.1:41172)
0927 02:45:25.672367 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:25.672378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:25.672434 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:25.672903 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:25.672904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:25.672906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:25.672952 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.672953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.673844 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.673845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.674779 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.674782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.676785 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.676786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.676851 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.676855 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:45:25.676904 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:45:25.676956 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:45:25.677042 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.672512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.0.98.3:37827)
0927 02:45:25.672721 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.672726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.672738 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:45:25.672777 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.672779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.672795 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.672901 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:25.672994 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.673773 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.674909 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.676703 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.676767 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.676834 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.676922 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.676962 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":122,"threads_started":1}
I20260927 02:45:25.677815 16689 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c (DEFAULT_TABLE table=test-table [id=f4872510787f4babb17ba1e2ba06e141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:45:25.678125 16689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:45:25.680850 16765 tablet_bootstrap.cc:492] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: Bootstrap starting.
I20260927 02:45:25.681438 16765 tablet_bootstrap.cc:654] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: Neither blocks nor log segments found. Creating new log.
I20260927 02:45:25.681924 16765 log.cc:824] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:25.682632 16765 tablet_bootstrap.cc:492] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: No bootstrap required, opened a new log
I20260927 02:45:25.682713 16765 ts_tablet_manager.cc:1397] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:45:25.684158 16765 raft_consensus.cc:348] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.684303 16765 raft_consensus.cc:374] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:45:25.684335 16765 raft_consensus.cc:729] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dcb9e13e3bc43d78677e866501b0c8a, State: Initialized, Role: FOLLOWER
I20260927 02:45:25.684482 16765 consensus_queue.cc:260] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.684773 16765 ts_tablet_manager.cc:1428] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:45:25.684803 16756 heartbeater.cc:503] Master 127.0.98.62:38417 was elected leader, sending a full tablet report...
W20260927 02:45:25.721390 16621 tablet.cc:2369] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:45:25.839852 16482 tablet.cc:2369] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:45:25.884586 16757 tablet.cc:2369] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:45:25.969480 16767 raft_consensus.cc:482] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:45:25.969600 16767 raft_consensus.cc:504] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.969939 16767 leader_election.cc:288] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707), 93fedfbe449346cd89d1a12daf219cb5 (127.0.98.1:43911)
I20260927 02:45:25.972734 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.970189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42693 (local address 127.0.98.2:44707)
0927 02:45:25.970244 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.970247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.970347 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:45:25.970412 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.970413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.970422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.970469 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:25.970599 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.971057 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.972042 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.972434 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.972487 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.972498 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.972630 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.972662 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:45:25.972822 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.970101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43911 (local address 127.0.98.3:38813)
0927 02:45:25.970216 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:25.970228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:25.970288 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:25.970540 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:25.970541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:25.970543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:25.970602 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.970603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.971251 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.971252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.971949 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.971951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.972558 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.972559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.972628 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.972631 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:45:25.972688 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:45:25.972732 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 02:45:25.972811 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.970286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38813 (local address 127.0.98.1:43911)
0927 02:45:25.970402 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:25.970404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:25.970411 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:45:25.970434 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:25.970434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:25.970455 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:25.970518 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:45:25.970667 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.971176 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.972039 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.972491 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.972552 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.972629 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:25.972702 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 02:45:25.972743 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260927 02:45:25.972746 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:25.970111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44707 (local address 127.0.98.3:42693)
0927 02:45:25.970273 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:25.970283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:25.970352 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:25.970477 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:25.970478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:25.970479 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:45:25.970546 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.970548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.971126 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.971128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.971969 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:25.971972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:25.972507 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:25.972508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:25.972576 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:25.972580 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:45:25.972620 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 02:45:25.972657 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":13,"thread_start_us":88,"threads_started":1}
I20260927 02:45:25.973083 16435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5cfe0e26e24a2fa0b4af242ee09e9c" candidate_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93fedfbe449346cd89d1a12daf219cb5" is_pre_election: true
I20260927 02:45:25.973239 16435 raft_consensus.cc:2454] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dcb9e13e3bc43d78677e866501b0c8a in term 0.
I20260927 02:45:25.973256 16574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5cfe0e26e24a2fa0b4af242ee09e9c" candidate_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" is_pre_election: true
I20260927 02:45:25.973405 16574 raft_consensus.cc:2454] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dcb9e13e3bc43d78677e866501b0c8a in term 0.
I20260927 02:45:25.973452 16644 leader_election.cc:302] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dcb9e13e3bc43d78677e866501b0c8a, 93fedfbe449346cd89d1a12daf219cb5; no voters:
I20260927 02:45:25.973562 16767 raft_consensus.cc:2781] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:45:25.973634 16767 raft_consensus.cc:482] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:45:25.973656 16767 raft_consensus.cc:3037] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:25.974330 16767 raft_consensus.cc:504] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.974447 16767 leader_election.cc:288] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [CANDIDATE]: Term 1 election: Requested vote from peers 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707), 93fedfbe449346cd89d1a12daf219cb5 (127.0.98.1:43911)
I20260927 02:45:25.974594 16574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5cfe0e26e24a2fa0b4af242ee09e9c" candidate_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dcaed5d633f41ac85e8d98d564ebb11"
I20260927 02:45:25.974640 16435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5cfe0e26e24a2fa0b4af242ee09e9c" candidate_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93fedfbe449346cd89d1a12daf219cb5"
I20260927 02:45:25.974699 16574 raft_consensus.cc:3037] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:25.974741 16435 raft_consensus.cc:3037] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:45:25.975528 16435 raft_consensus.cc:2454] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dcb9e13e3bc43d78677e866501b0c8a in term 1.
I20260927 02:45:25.975590 16574 raft_consensus.cc:2454] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dcb9e13e3bc43d78677e866501b0c8a in term 1.
I20260927 02:45:25.975742 16645 leader_election.cc:302] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dcb9e13e3bc43d78677e866501b0c8a, 8dcaed5d633f41ac85e8d98d564ebb11; no voters:
I20260927 02:45:25.975883 16767 raft_consensus.cc:2781] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:45:25.976029 16767 raft_consensus.cc:686] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 LEADER]: Becoming Leader. State: Replica: 3dcb9e13e3bc43d78677e866501b0c8a, State: Running, Role: LEADER
I20260927 02:45:25.976114 16767 consensus_queue.cc:237] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:25.976892 16293 catalog_manager.cc:5495] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } health_report { overall_health: UNKNOWN } } }
I20260927 02:45:26.031399 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:26.029267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.0.98.3:37827)
0927 02:45:26.029321 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:26.029323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:26.029330 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:45:26.029365 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:26.029365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:26.029386 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:26.029446 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:45:26.029536 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:26.030007 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:26.030713 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:26.030915 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:26.030975 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:26.031094 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:45:26.031194 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:45:26.031251 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:26.031268 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 02:45:26.031287 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:45:26.033281 16435 raft_consensus.cc:1261] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Refusing update from remote peer 3dcb9e13e3bc43d78677e866501b0c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:45:26.033281 16574 raft_consensus.cc:1261] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Refusing update from remote peer 3dcb9e13e3bc43d78677e866501b0c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:45:26.033640 16770 consensus_queue.cc:1035] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:45:26.033782 16767 consensus_queue.cc:1035] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:45:26.035681 16772 mvcc.cc:205] Tried to move back new op lower bound from 7333794308229300224 to 7333794307998658560. Current Snapshot: MvccSnapshot[applied={T|T < 7333794308229300224}]
I20260927 02:45:26.035935 16773 mvcc.cc:205] Tried to move back new op lower bound from 7333794308229300224 to 7333794307998658560. Current Snapshot: MvccSnapshot[applied={T|T < 7333794308229300224}]
I20260927 02:45:26.035949 16774 mvcc.cc:205] Tried to move back new op lower bound from 7333794308229300224 to 7333794307998658560. Current Snapshot: MvccSnapshot[applied={T|T < 7333794308229300224}]
I20260927 02:45:26.037909 16292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:45:26.037997 16292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:45:26.038676 16292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Sending DeleteTablet for 3 replicas of tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c
I20260927 02:45:26.038940 16554 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53940
I20260927 02:45:26.038964 16689 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41172
I20260927 02:45:26.039233 16413 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:45:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59056
I20260927 02:45:26.039258 16783 tablet_replica.cc:330] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: stopping tablet replica
I20260927 02:45:26.039315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 16331
I20260927 02:45:26.039891 16783 raft_consensus.cc:2229] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:45:26.040009 16783 raft_consensus.cc:2258] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:45:26.040220 16782 tablet_replica.cc:330] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: stopping tablet replica
I20260927 02:45:26.040314 16782 raft_consensus.cc:2229] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:45:26.040444 16782 raft_consensus.cc:2258] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:45:26.040694 16783 ts_tablet_manager.cc:1905] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:45:26.042315 16783 ts_tablet_manager.cc:1918] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:45:26.042377 16783 log.cc:1196] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/9a5cfe0e26e24a2fa0b4af242ee09e9c
I20260927 02:45:26.042579 16783 ts_tablet_manager.cc:1939] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 8dcaed5d633f41ac85e8d98d564ebb11: Deleting consensus metadata
I20260927 02:45:26.043152 16279 catalog_manager.cc:4841] TS 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707): tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c (table test-table [id=f4872510787f4babb17ba1e2ba06e141]) successfully deleted
W20260927 02:45:26.044255 16278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43911 (error 108)
I20260927 02:45:26.044422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 16486
I20260927 02:45:26.045469 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:26.045358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43911 (local address 127.0.0.1:59062)
0927 02:45:26.045404 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:26.045439 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:45:26.045576 16278 catalog_manager.cc:4584] TS 93fedfbe449346cd89d1a12daf219cb5 (127.0.98.1:43911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43911: connect: Connection refused (error 111)
I20260927 02:45:26.047349 16782 ts_tablet_manager.cc:1905] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:45:26.049325 16782 ts_tablet_manager.cc:1918] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:45:26.049408 16782 log.cc:1196] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/9a5cfe0e26e24a2fa0b4af242ee09e9c
I20260927 02:45:26.049676 16782 ts_tablet_manager.cc:1939] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 3dcb9e13e3bc43d78677e866501b0c8a: Deleting consensus metadata
I20260927 02:45:26.050133 16277 catalog_manager.cc:4841] TS 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827): tablet 9a5cfe0e26e24a2fa0b4af242ee09e9c (table test-table [id=f4872510787f4babb17ba1e2ba06e141]) successfully deleted
I20260927 02:45:26.051457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 16623
I20260927 02:45:26.055589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 16260
I20260927 02:45:46.060369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--webserver_interface=127.0.98.62
--webserver_port=34531
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:45:46.141597 16786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:46.141834 16786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:46.141855 16786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:46.143095 16786 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:45:46.143139 16786 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:45:46.143151 16786 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:45:46.143162 16786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:45:46.143173 16786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:46.143184 16786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:46.143198 16786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:45:46.143221 16786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:45:46.144812 16786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:46.145064 16786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:46.145293 16786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:45:46.150175 16786 server_base.cc:1034] running on GCE node
W20260927 02:45:46.150142 16791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:46.150134 16794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:46.150157 16792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:46.150707 16786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:46.151010 16786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:46.152177 16786 hybrid_clock.cc:648] HybridClock initialized: now 1790477146152154 us; error 37 us; skew 500 ppm
Sep 27 02:45:46 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:46.154798 16786 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:45:46.155258 16786 webserver.cc:460] Webserver started at http://127.0.98.62:34531/ using document root <none> and password file <none>
I20260927 02:45:46.155447 16786 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:46.155485 16786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:46.156838 16786 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 02:45:46.157743 16801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:46.157943 16786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:46.158033 16786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "54557d141cc74e43bab5bb488bb58ced"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "e813bba08ed1d2fc2f9af81045e0ea92"
server_key_iv: "3a369018bc3b1384f55a13a8f4c128f7"
server_key_version: "encryptionkey@0"
I20260927 02:45:46.158387 16786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:46.170455 16786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:46.172684 16786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:46.172806 16786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:46.176517 16786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38417
I20260927 02:45:46.176564 16853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38417 every 8 connection(s)
I20260927 02:45:46.176913 16786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:45:46.178423 16854 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:45:46.179073 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Bootstrap starting.
I20260927 02:45:46.185592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 16786
I20260927 02:45:46.185892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43911
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38417
--webserver_port=43753
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:45:46.187141 16854 log.cc:824] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:46.189373 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:45:46.189638 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Bootstrap complete.
I20260927 02:45:46.191666 16854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:46.191890 16854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54557d141cc74e43bab5bb488bb58ced, State: Initialized, Role: FOLLOWER
I20260927 02:45:46.192041 16854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:46.192114 16854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:45:46.192162 16854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:45:46.192207 16854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:45:46.193226 16854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:46.193326 16854 leader_election.cc:302] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54557d141cc74e43bab5bb488bb58ced; no voters:
I20260927 02:45:46.193490 16854 leader_election.cc:288] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:45:46.193547 16858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:45:46.193683 16858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [term 3 LEADER]: Becoming Leader. State: Replica: 54557d141cc74e43bab5bb488bb58ced, State: Running, Role: LEADER
I20260927 02:45:46.193753 16854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:45:46.193770 16858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260927 02:45:46.194087 16859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "54557d141cc74e43bab5bb488bb58ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260927 02:45:46.194208 16859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: This master's current role is: LEADER
I20260927 02:45:46.194453 16860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54557d141cc74e43bab5bb488bb58ced. Latest consensus state: current_term: 3 leader_uuid: "54557d141cc74e43bab5bb488bb58ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54557d141cc74e43bab5bb488bb58ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260927 02:45:46.194553 16860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced [sys.catalog]: This master's current role is: LEADER
I20260927 02:45:46.194563 16866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:45:46.195393 16866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:45:46.195492 16866 catalog_manager.cc:1259] Loaded cluster ID: 2df93773bcaa404090f78fb718754a8e
I20260927 02:45:46.195533 16866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:45:46.196363 16866 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:45:46.196559 16866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Loaded TSK: 0
I20260927 02:45:46.196596 16866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 54557d141cc74e43bab5bb488bb58ced: Loaded TSK: 1
F20260927 02:45:46.196822 16866 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f00f71369ad google::LogMessage::Fail() at ??:0
@ 0x7f00f713988a google::LogMessage::SendToLog() at ??:0
@ 0x7f00f71364de google::LogMessage::Flush() at ??:0
@ 0x7f00f7137a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f00f98a3104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f00f98a0d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f00ff0991cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f00ff09948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f00ff099594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f00ff05fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f00ff069ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f00f90040d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f00f8ff9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f00f9e5d6db start_thread at ??:0
@ 0x7f00f5c3461f clone at ??:0
W20260927 02:45:46.301618 16856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:46.301821 16856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:46.301846 16856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:46.303256 16856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:46.303313 16856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:46.303341 16856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:45:46.305009 16856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43753
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:46.305241 16856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:46.305461 16856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:46.311456 16896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:46.311496 16895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:46.311492 16898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:46.311666 16856 server_base.cc:1034] running on GCE node
I20260927 02:45:46.311969 16856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:46.312288 16856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:46.313476 16856 hybrid_clock.cc:648] HybridClock initialized: now 1790477146313456 us; error 33 us; skew 500 ppm
Sep 27 02:45:46 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:46.316547 16856 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:45:46.316999 16856 webserver.cc:460] Webserver started at http://127.0.98.1:43753/ using document root <none> and password file <none>
I20260927 02:45:46.317207 16856 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:46.317257 16856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:46.318601 16856 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:46.319259 16905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:46.319435 16856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:46.319542 16856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "93fedfbe449346cd89d1a12daf219cb5"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "f6e3b36cc39c952b1e8407a0783bddef"
server_key_iv: "896628e08b3bd800971ea86e7260afb4"
server_key_version: "encryptionkey@0"
I20260927 02:45:46.319877 16856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:46.342559 16856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:46.344767 16856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:46.344907 16856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:46.345134 16856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:46.345563 16912 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:45:46.347513 16856 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:45:46.347575 16856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:45:46.347611 16856 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:45:46.348273 16856 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:45:46.348306 16856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:45:46.348419 16912 tablet_bootstrap.cc:492] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Bootstrap starting.
I20260927 02:45:46.354768 16856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43911
I20260927 02:45:46.354916 17019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43911 every 8 connection(s)
I20260927 02:45:46.355190 16856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:45:46.359233 16912 log.cc:824] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:45:46.361271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 16856
I20260927 02:45:46.361616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44707
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38417
--webserver_port=41681
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:45:46.362028 16912 tablet_bootstrap.cc:492] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:45:46.362315 16912 tablet_bootstrap.cc:492] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Bootstrap complete.
I20260927 02:45:46.363380 16912 ts_tablet_manager.cc:1397] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260927 02:45:46.364931 16912 raft_consensus.cc:348] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:46.365270 16912 raft_consensus.cc:729] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93fedfbe449346cd89d1a12daf219cb5, State: Initialized, Role: FOLLOWER
I20260927 02:45:46.365454 16912 consensus_queue.cc:260] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:46.365782 16912 ts_tablet_manager.cc:1428] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260927 02:45:46.403440 17022 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:45:46.355838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:43275)
0927 02:45:46.355998 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:46.356057 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:46.356588 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:46.403228 (+ 46640us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
W20260927 02:45:46.403724 17020 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38417 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38417: Client connection negotiation failed: client connection to 127.0.98.62:38417: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:45:46.404057 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:46.403935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:42813)
0927 02:45:46.403985 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:46.404026 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:45:46.404425 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:46.404331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:56083)
0927 02:45:46.404364 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:46.404407 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:45:46.404523 17020 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:45:46.444151 17024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:46.444355 17024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:46.444378 17024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:46.445542 17024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:46.445582 17024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:46.445606 17024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:45:46.447069 17024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41681
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:46.447260 17024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:46.447431 17024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:46.452306 17033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:46.452296 17031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:46.452322 17030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:46.452591 17024 server_base.cc:1034] running on GCE node
I20260927 02:45:46.452878 17024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:46.453115 17024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:46.454259 17024 hybrid_clock.cc:648] HybridClock initialized: now 1790477146454246 us; error 29 us; skew 500 ppm
Sep 27 02:45:46 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:46.456633 17024 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:45:46.457080 17024 webserver.cc:460] Webserver started at http://127.0.98.2:41681/ using document root <none> and password file <none>
I20260927 02:45:46.457283 17024 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:46.457327 17024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:46.458632 17024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:46.459295 17040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:46.459456 17024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:46.459539 17024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "8dcaed5d633f41ac85e8d98d564ebb11"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "528ecab025cfbf3225adf5b1e673703f"
server_key_iv: "143c5da8f0603ab842ef8aeb753cd4bf"
server_key_version: "encryptionkey@0"
I20260927 02:45:46.459903 17024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:46.466884 17024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:46.469139 17024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:46.469256 17024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:46.469503 17024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:46.469792 17024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:45:46.469825 17024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:46.469846 17024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:45:46.469869 17024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:46.475404 17024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44707
I20260927 02:45:46.475447 17153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44707 every 8 connection(s)
I20260927 02:45:46.475782 17024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:45:46.476902 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:46.476450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:43407)
0927 02:45:46.476658 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:46.476757 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
W20260927 02:45:46.477274 17154 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38417 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38417: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
I20260927 02:45:46.477499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 17024
I20260927 02:45:46.477617 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:46.477507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:54147)
0927 02:45:46.477559 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:46.477594 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:45:46.477785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37827
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38417
--webserver_port=44817
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:45:46.477922 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:46.477829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:34237)
0927 02:45:46.477877 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:46.477904 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:45:46.478063 17154 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:45:46.557893 17157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:45:46.558099 17157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:45:46.558120 17157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:45:46.559201 17157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:45:46.559233 17157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:45:46.559257 17157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:45:46.560631 17157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44817
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:45:46.560817 17157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:45:46.560997 17157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:45:46.565805 17162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:46.565826 17165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:45:46.565840 17163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:45:46.566100 17157 server_base.cc:1034] running on GCE node
I20260927 02:45:46.566279 17157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:45:46.566488 17157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:45:46.567646 17157 hybrid_clock.cc:648] HybridClock initialized: now 1790477146567633 us; error 25 us; skew 500 ppm
Sep 27 02:45:46 dist-test-slave-44kr krb5kdc[15716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:45:46.570261 17157 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:45:46.570732 17157 webserver.cc:460] Webserver started at http://127.0.98.3:44817/ using document root <none> and password file <none>
I20260927 02:45:46.570947 17157 fs_manager.cc:359] Metadata directory not provided
I20260927 02:45:46.571004 17157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:45:46.572420 17157 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:46.573081 17172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:46.573269 17157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:45:46.573354 17157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "3dcb9e13e3bc43d78677e866501b0c8a"
format_stamp: "Formatted at 2026-09-27 02:45:04 on dist-test-slave-44kr"
server_key: "8297c1c40ee6607d04a6fb0335fc8f98"
server_key_iv: "62a058faec4fbb834b61c7f5e1eb29ed"
server_key_version: "encryptionkey@0"
I20260927 02:45:46.573701 17157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:45:46.588328 17157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:45:46.590698 17157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:45:46.590855 17157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:45:46.591137 17157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:45:46.591434 17157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:45:46.591466 17157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:46.591497 17157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:45:46.591517 17157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:45:46.597013 17157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37827
I20260927 02:45:46.597067 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37827 every 8 connection(s)
I20260927 02:45:46.597383 17157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:45:46.598409 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:46.598003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:49351)
0927 02:45:46.598190 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:46.598267 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
W20260927 02:45:46.598795 17286 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38417 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38417: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
I20260927 02:45:46.599073 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:46.598977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:35149)
0927 02:45:46.599025 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:46.599055 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:45:46.599303 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:46.599242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:49725)
0927 02:45:46.599273 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:46.599291 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:45:46.599383 17286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:45:46.602829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 17157
I20260927 02:45:47.405254 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:47.404928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:51701)
0927 02:45:47.405139 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:47.405186 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":115,"threads_started":1}
I20260927 02:45:47.478701 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:47.478453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:40123)
0927 02:45:47.478599 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:47.478644 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260927 02:45:47.599956 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:47.599766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:51383)
0927 02:45:47.599871 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:47.599906 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260927 02:45:47.968686 17293 raft_consensus.cc:482] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:45:47.968811 17293 raft_consensus.cc:504] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:47.969118 17293 leader_election.cc:288] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827), 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707)
Sep 27 02:45:47 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 02:45:47 dist-test-slave-44kr krb5kdc[15716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 02:45:47.979382 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:47.969396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52069 (local address 127.0.98.2:44707)
0927 02:45:47.969539 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:47.969549 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:47.969755 (+ 206us) server_negotiation.cc:407] Connection header received
0927 02:45:47.970266 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:47.970269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:47.970309 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:47.970445 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:47.971155 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:47.971994 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:47.973001 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:47.973389 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:47.973484 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:47.975761 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:47.975765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:47.975766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:47.975780 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:47.977591 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:47.978058 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:47.978062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:47.978062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:47.978165 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:47.978401 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:47.978402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:47.978402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:47.978733 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:47.978792 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:47.979141 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 02:45:47.979279 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":51,"threads_started":1}
I20260927 02:45:47.979382 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:47.969311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44707 (local address 127.0.98.1:52069)
0927 02:45:47.969639 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:47.969650 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:47.969707 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:47.970441 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:47.970442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:47.970845 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:47.971041 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:47.971046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:47.972089 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:47.972090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:47.972924 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:47.972926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:47.973478 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:47.973479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:47.973580 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:47.974150 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:47.974154 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:47.975596 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:47.977670 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:47.977678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:47.977681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:47.977962 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:47.978239 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:47.978240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:47.978240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:47.978341 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:47.978824 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:47.978832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:47.978974 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 02:45:47.979079 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 02:45:47.979270 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":20,"thread_start_us":54,"threads_started":1}
I20260927 02:45:47.979517 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:47.969499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37289 (local address 127.0.98.3:37827)
0927 02:45:47.969632 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:45:47.969637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:45:47.969660 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:45:47.970197 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:45:47.970201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:45:47.970246 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:45:47.970358 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:45:47.971155 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:47.971999 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:47.973011 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:45:47.973389 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:45:47.973485 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:47.976124 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:45:47.976127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:45:47.976127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:45:47.976137 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:45:47.977926 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:47.978379 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:47.978383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:47.978383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:47.978468 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:45:47.978685 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:45:47.978686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:45:47.978686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:45:47.978984 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:45:47.979080 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:45:47.979289 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 02:45:47.979418 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260927 02:45:47.979457 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:47.969311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37827 (local address 127.0.98.1:37289)
0927 02:45:47.969420 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:47.969431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:45:47.969490 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:45:47.970438 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:45:47.970441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:45:47.970847 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:45:47.971041 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:47.971046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:47.972099 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:47.972100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:47.972925 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:45:47.972927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:45:47.973467 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:45:47.973468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:45:47.973579 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:45:47.974100 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:45:47.974108 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:45:47.976004 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:45:47.978035 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:47.978037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:47.978037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:47.978306 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:47.978529 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:45:47.978530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:45:47.978530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:45:47.978595 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:45:47.979069 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:45:47.979070 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:45:47.979118 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 02:45:47.979198 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:45:47.979343 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":98,"thread_start_us":48,"threads_started":1}
W20260927 02:45:47.980662 16906 leader_election.cc:341] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827): Not found: Tablet not found: 9a5cfe0e26e24a2fa0b4af242ee09e9c
W20260927 02:45:47.980748 16908 leader_election.cc:341] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707): Not found: Tablet not found: 9a5cfe0e26e24a2fa0b4af242ee09e9c
I20260927 02:45:47.980825 16908 leader_election.cc:302] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93fedfbe449346cd89d1a12daf219cb5; no voters: 3dcb9e13e3bc43d78677e866501b0c8a, 8dcaed5d633f41ac85e8d98d564ebb11
I20260927 02:45:47.980935 17293 raft_consensus.cc:2726] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:45:48.405953 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:48.405801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:52135)
0927 02:45:48.405875 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:48.405909 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 02:45:48.479426 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:48.479166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:36673)
0927 02:45:48.479325 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:48.479363 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260927 02:45:48.600682 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:48.600453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:42549)
0927 02:45:48.600601 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:48.600641 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260927 02:45:49.406623 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:49.406425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:43811)
0927 02:45:49.406542 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:49.406577 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260927 02:45:49.480086 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:49.479879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:52615)
0927 02:45:49.479998 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:49.480041 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260927 02:45:49.601348 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:49.601108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:38725)
0927 02:45:49.601255 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:49.601307 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260927 02:45:49.745208 17303 raft_consensus.cc:482] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:45:49.745312 17303 raft_consensus.cc:504] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:49.745476 17303 leader_election.cc:288] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827), 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707)
W20260927 02:45:49.745958 16906 leader_election.cc:341] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827): Not found: Tablet not found: 9a5cfe0e26e24a2fa0b4af242ee09e9c
W20260927 02:45:49.746048 16908 leader_election.cc:341] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707): Not found: Tablet not found: 9a5cfe0e26e24a2fa0b4af242ee09e9c
I20260927 02:45:49.746090 16908 leader_election.cc:302] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93fedfbe449346cd89d1a12daf219cb5; no voters: 3dcb9e13e3bc43d78677e866501b0c8a, 8dcaed5d633f41ac85e8d98d564ebb11
I20260927 02:45:49.746156 17303 raft_consensus.cc:2726] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:45:50.407469 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:50.407196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:42391)
0927 02:45:50.407370 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:50.407420 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
I20260927 02:45:50.480816 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:50.480564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:57977)
0927 02:45:50.480727 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:50.480771 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260927 02:45:50.602095 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:50.601845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:48081)
0927 02:45:50.602005 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:50.602050 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260927 02:45:51.408331 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:51.408021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:52793)
0927 02:45:51.408228 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:51.408279 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":136,"threads_started":1}
I20260927 02:45:51.481581 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:51.481339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:37859)
0927 02:45:51.481483 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:51.481532 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260927 02:45:51.602921 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:51.602648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:59199)
0927 02:45:51.602827 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:51.602873 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260927 02:45:52.409211 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:52.408927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:55027)
0927 02:45:52.409098 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:52.409160 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260927 02:45:52.482458 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:52.482126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:44871)
0927 02:45:52.482363 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:52.482409 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":126,"threads_started":1}
I20260927 02:45:52.603645 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:52.603410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:34683)
0927 02:45:52.603562 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:52.603600 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260927 02:45:53.118353 17313 raft_consensus.cc:482] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:45:53.118469 17313 raft_consensus.cc:504] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:53.118649 17313 leader_election.cc:288] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827), 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707)
W20260927 02:45:53.119092 16908 leader_election.cc:341] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707): Not found: Tablet not found: 9a5cfe0e26e24a2fa0b4af242ee09e9c
W20260927 02:45:53.119191 16906 leader_election.cc:341] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827): Not found: Tablet not found: 9a5cfe0e26e24a2fa0b4af242ee09e9c
I20260927 02:45:53.119225 16906 leader_election.cc:302] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93fedfbe449346cd89d1a12daf219cb5; no voters: 3dcb9e13e3bc43d78677e866501b0c8a, 8dcaed5d633f41ac85e8d98d564ebb11
I20260927 02:45:53.119292 17313 raft_consensus.cc:2726] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:45:53.410002 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:53.409761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:39151)
0927 02:45:53.409909 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:53.409953 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260927 02:45:53.483275 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:53.483032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:48121)
0927 02:45:53.483185 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:53.483231 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260927 02:45:53.604512 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:53.604219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:59907)
0927 02:45:53.604405 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:53.604461 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260927 02:45:54.410753 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:54.410505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:52079)
0927 02:45:54.410669 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:54.410707 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260927 02:45:54.483978 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:54.483788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:54101)
0927 02:45:54.483904 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:54.483931 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260927 02:45:54.605335 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:54.605062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:32999)
0927 02:45:54.605252 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:54.605293 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260927 02:45:55.093401 17320 raft_consensus.cc:482] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:45:55.093515 17320 raft_consensus.cc:504] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dcb9e13e3bc43d78677e866501b0c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37827 } } peers { permanent_uuid: "8dcaed5d633f41ac85e8d98d564ebb11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "93fedfbe449346cd89d1a12daf219cb5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43911 } }
I20260927 02:45:55.093705 17320 leader_election.cc:288] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827), 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707)
W20260927 02:45:55.094322 16906 leader_election.cc:341] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dcb9e13e3bc43d78677e866501b0c8a (127.0.98.3:37827): Not found: Tablet not found: 9a5cfe0e26e24a2fa0b4af242ee09e9c
W20260927 02:45:55.094414 16908 leader_election.cc:341] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8dcaed5d633f41ac85e8d98d564ebb11 (127.0.98.2:44707): Not found: Tablet not found: 9a5cfe0e26e24a2fa0b4af242ee09e9c
I20260927 02:45:55.094480 16908 leader_election.cc:302] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93fedfbe449346cd89d1a12daf219cb5; no voters: 3dcb9e13e3bc43d78677e866501b0c8a, 8dcaed5d633f41ac85e8d98d564ebb11
I20260927 02:45:55.094600 17320 raft_consensus.cc:2726] T 9a5cfe0e26e24a2fa0b4af242ee09e9c P 93fedfbe449346cd89d1a12daf219cb5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:45:55.411504 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:55.411287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:58583)
0927 02:45:55.411412 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:55.411453 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260927 02:45:55.484700 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:55.484485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:58909)
0927 02:45:55.484611 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:55.484655 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260927 02:45:55.606148 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:55.605857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:44439)
0927 02:45:55.606015 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:55.606060 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260927 02:45:56.412349 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:56.412103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:50259)
0927 02:45:56.412266 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:56.412302 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260927 02:45:56.485463 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:56.485209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:36637)
0927 02:45:56.485369 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:56.485416 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260927 02:45:56.606935 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:56.606698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:49307)
0927 02:45:56.606841 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:56.606887 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260927 02:45:57.413123 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:57.412805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:59955)
0927 02:45:57.413021 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:57.413073 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":65,"threads_started":1}
I20260927 02:45:57.486219 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:57.485988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:58565)
0927 02:45:57.486129 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:57.486171 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260927 02:45:57.607719 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:57.607499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:37657)
0927 02:45:57.607625 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:57.607671 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260927 02:45:58.413913 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:58.413704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:47499)
0927 02:45:58.413820 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:58.413865 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260927 02:45:58.487041 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:58.486811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:42187)
0927 02:45:58.486946 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:58.486992 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260927 02:45:58.608474 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:58.608225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:53051)
0927 02:45:58.608354 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:58.608419 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260927 02:45:59.414680 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:59.414411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:53519)
0927 02:45:59.414572 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:59.414618 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260927 02:45:59.487764 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:59.487519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:36413)
0927 02:45:59.487673 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:59.487716 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260927 02:45:59.609257 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:45:59.608999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:46939)
0927 02:45:59.609162 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:45:59.609208 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260927 02:46:00.415459 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:00.415226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:33413)
0927 02:46:00.415366 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:00.415410 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260927 02:46:00.488569 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:00.488286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:42231)
0927 02:46:00.488474 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:00.488520 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":65,"threads_started":1}
I20260927 02:46:00.610083 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:00.609804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:48793)
0927 02:46:00.609956 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:00.609994 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260927 02:46:01.416237 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:01.415976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:59251)
0927 02:46:01.416156 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:01.416191 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260927 02:46:01.489295 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:01.489099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:60017)
0927 02:46:01.489218 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:01.489249 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260927 02:46:01.604084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 16856
I20260927 02:46:01.608145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 17024
I20260927 02:46:01.610759 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:01.610584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:54915)
0927 02:46:01.610690 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:01.610719 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
I20260927 02:46:01.611716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 17157
2026-09-27T02:46:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58597 ms)
[----------] 30 tests from SecurityITest (232364 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:01 dist-test-slave-44kr krb5kdc[17344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:01 dist-test-slave-44kr krb5kdc[17344](info): set up 2 sockets
Sep 27 02:46:01 dist-test-slave-44kr krb5kdc[17344](info): commencing operation
krb5kdc: starting...
W20260927 02:46:02.583954 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:02 dist-test-slave-44kr krb5kdc[17344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:02.697441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:02.777745 17357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:02.777973 17357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:02.778005 17357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:02.779176 17357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:02.779220 17357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:02.779239 17357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:02.779255 17357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:02.779269 17357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:02.780807 17357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:02.781067 17357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:02.781281 17357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:02.786511 17363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:02.786541 17362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:02.786511 17365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:02.786543 17357 server_base.cc:1034] running on GCE node
I20260927 02:46:02.786991 17357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:02.787283 17357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:02.788478 17357 hybrid_clock.cc:648] HybridClock initialized: now 1790477162788458 us; error 39 us; skew 500 ppm
Sep 27 02:46:02 dist-test-slave-44kr krb5kdc[17344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:02.790904 17357 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:02.791333 17357 webserver.cc:460] Webserver started at http://127.0.98.62:45453/ using document root <none> and password file <none>
I20260927 02:46:02.791524 17357 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:02.791566 17357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:02.791658 17357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:02.792711 17357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "0a09b6abfe814649a06e59aced323edf"
format_stamp: "Formatted at 2026-09-27 02:46:02 on dist-test-slave-44kr"
server_key: "98230e687855ccd8403ca87ace2cf96e"
server_key_iv: "3036b08a4e00e452d07cb2b1966271aa"
server_key_version: "encryptionkey@0"
I20260927 02:46:02.793017 17357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a09b6abfe814649a06e59aced323edf"
format_stamp: "Formatted at 2026-09-27 02:46:02 on dist-test-slave-44kr"
server_key: "98230e687855ccd8403ca87ace2cf96e"
server_key_iv: "3036b08a4e00e452d07cb2b1966271aa"
server_key_version: "encryptionkey@0"
I20260927 02:46:02.794194 17357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:46:02.794886 17372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:02.795156 17357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:02.795228 17357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "0a09b6abfe814649a06e59aced323edf"
format_stamp: "Formatted at 2026-09-27 02:46:02 on dist-test-slave-44kr"
server_key: "98230e687855ccd8403ca87ace2cf96e"
server_key_iv: "3036b08a4e00e452d07cb2b1966271aa"
server_key_version: "encryptionkey@0"
I20260927 02:46:02.795279 17357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:02.819775 17357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:02.821900 17357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:02.822024 17357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:02.825907 17357 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46297
I20260927 02:46:02.826013 17424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46297 every 8 connection(s)
I20260927 02:46:02.826293 17357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:02.827162 17425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:02.829447 17425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf: Bootstrap starting.
I20260927 02:46:02.829993 17425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:02.830255 17425 log.cc:824] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:02.830976 17425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf: No bootstrap required, opened a new log
I20260927 02:46:02.832327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 17357
I20260927 02:46:02.832445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:02.832655 392 external_mini_cluster.cc:1428] Setting key b2092442527fe6f26a168250e406d344
I20260927 02:46:02.832868 17425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09b6abfe814649a06e59aced323edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } }
I20260927 02:46:02.833029 17425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:02.833063 17425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a09b6abfe814649a06e59aced323edf, State: Initialized, Role: FOLLOWER
I20260927 02:46:02.833201 17425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09b6abfe814649a06e59aced323edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } }
I20260927 02:46:02.833312 17425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:02.833360 17425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:02.833410 17425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:02.834230 17425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09b6abfe814649a06e59aced323edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } }
I20260927 02:46:02.834319 17425 leader_election.cc:302] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a09b6abfe814649a06e59aced323edf; no voters:
I20260927 02:46:02.834465 17425 leader_election.cc:288] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:02.834493 17430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:02.834658 17430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [term 1 LEADER]: Becoming Leader. State: Replica: 0a09b6abfe814649a06e59aced323edf, State: Running, Role: LEADER
I20260927 02:46:02.834740 17425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:02.834764 17430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09b6abfe814649a06e59aced323edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } }
I20260927 02:46:02.835103 17430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a09b6abfe814649a06e59aced323edf. Latest consensus state: current_term: 1 leader_uuid: "0a09b6abfe814649a06e59aced323edf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09b6abfe814649a06e59aced323edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } }
I20260927 02:46:02.835355 17430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:02.835278 17431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a09b6abfe814649a06e59aced323edf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a09b6abfe814649a06e59aced323edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } }
I20260927 02:46:02.835943 17431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf [sys.catalog]: This master's current role is: LEADER
Sep 27 02:46:02 dist-test-slave-44kr krb5kdc[17344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:02.835793 17439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:02.843211 17439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:02.843755 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:02.833135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.0.98.62:46297)
0927 02:46:02.833311 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:02.833321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:02.833361 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:46:02.833510 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:02.833516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:02.833559 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:02.833689 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:02.834294 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:02.835634 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:02.836856 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:02.837451 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:02.839119 (+ 1668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:02.839963 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:02.839967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:02.839968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:02.839979 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:02.842012 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:02.842389 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:02.842392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:02.842392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:02.842503 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:02.842689 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:02.842689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:02.842690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:02.843063 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:02.843198 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:02.843394 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 02:46:02.843544 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260927 02:46:02.844983 17439 catalog_manager.cc:1347] Generated new cluster ID: 91506abda0f9469b95c2bd5606480a62
I20260927 02:46:02.845036 17439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:02.855693 17439 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:02.856530 17439 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:02.865365 17439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf: Generated new TSK 0
I20260927 02:46:02.865569 17439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:46:02.896191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46297
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:03.009016 17451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:03.009245 17451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:03.009271 17451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:03.010458 17451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:03.010512 17451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:03.010541 17451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:03.012074 17451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:03.012326 17451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:03.012600 17451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:03.017556 17459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:03.017557 17456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:03.017611 17457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:03.017679 17451 server_base.cc:1034] running on GCE node
I20260927 02:46:03.017992 17451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:03.018267 17451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:03.019445 17451 hybrid_clock.cc:648] HybridClock initialized: now 1790477163019427 us; error 37 us; skew 500 ppm
Sep 27 02:46:03 dist-test-slave-44kr krb5kdc[17344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:03.022083 17451 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:46:03.022486 17451 webserver.cc:460] Webserver started at http://127.0.98.1:38943/ using document root <none> and password file <none>
I20260927 02:46:03.022678 17451 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:03.022722 17451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:03.022843 17451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:03.023852 17451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "6b4da3b7c82640d286e93865348834f9"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "1069604d7b594fb59c113cb3fb447cfc"
server_key_iv: "1d929edf693c201386b27e7022eb6b27"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.024175 17451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6b4da3b7c82640d286e93865348834f9"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "1069604d7b594fb59c113cb3fb447cfc"
server_key_iv: "1d929edf693c201386b27e7022eb6b27"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.025482 17451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:03.026261 17466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.026412 17451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:03.026507 17451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "6b4da3b7c82640d286e93865348834f9"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "1069604d7b594fb59c113cb3fb447cfc"
server_key_iv: "1d929edf693c201386b27e7022eb6b27"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.026572 17451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:03.033879 17451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:03.036115 17451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:03.036250 17451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:03.036495 17451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:03.036803 17451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:03.036837 17451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.036866 17451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:03.036890 17451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.042526 17451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34779
I20260927 02:46:03.042562 17579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34779 every 8 connection(s)
I20260927 02:46:03.042866 17451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:46:03 dist-test-slave-44kr krb5kdc[17344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:03.051990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 17451
I20260927 02:46:03.052121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:46:03.052330 392 external_mini_cluster.cc:1428] Setting key 3a434a675173659fb63b1699d16e56d6
I20260927 02:46:03.052662 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.043625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46601 (local address 127.0.98.62:46297)
0927 02:46:03.043676 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.043678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.043876 (+ 198us) server_negotiation.cc:407] Connection header received
0927 02:46:03.044849 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.044851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.044864 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.044909 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:03.045538 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.046062 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.047106 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.047308 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.047373 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.049575 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:03.049576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:03.049577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:03.049584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:03.050951 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:03.051382 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:03.051383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:03.051383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:03.051477 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:03.051824 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:03.051825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:03.051825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:03.051997 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:03.052084 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.052473 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.052587 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:46:03.052858 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.043591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46297 (local address 127.0.98.1:46601)
0927 02:46:03.043753 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.043798 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:03.044735 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:03.044975 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:03.044979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:03.045245 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:03.045436 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.045440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.046140 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.046141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.047000 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.047002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.047385 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.047386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.047498 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.048007 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:03.048016 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:03.049480 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:03.051050 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:03.051056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:03.051059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:03.051291 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:03.051570 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:03.051571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:03.051571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:03.051725 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:03.052083 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:03.052088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:03.052260 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 02:46:03.052367 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 02:46:03.052557 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260927 02:46:03.053380 17580 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46297
I20260927 02:46:03.053488 17580 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:03.053702 17580 heartbeater.cc:511] Master 127.0.98.62:46297 requested a full tablet report, sending...
I20260927 02:46:03.054247 17389 ts_manager.cc:194] Registered new tserver with Master: 6b4da3b7c82640d286e93865348834f9 (127.0.98.1:34779)
I20260927 02:46:03.054975 17389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46601
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:46:03.077549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46297
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:03.157682 17585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:03.157879 17585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:03.157902 17585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:03.159062 17585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:03.159106 17585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:03.159132 17585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:03.160647 17585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:03.160866 17585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:03.161072 17585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:03.166270 17591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:03.166245 17590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:03.166272 17585 server_base.cc:1034] running on GCE node
W20260927 02:46:03.166245 17593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:03.166740 17585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:03.166981 17585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:03.168154 17585 hybrid_clock.cc:648] HybridClock initialized: now 1790477163168124 us; error 44 us; skew 500 ppm
Sep 27 02:46:03 dist-test-slave-44kr krb5kdc[17344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:03.170670 17585 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:46:03.171175 17585 webserver.cc:460] Webserver started at http://127.0.98.2:33639/ using document root <none> and password file <none>
I20260927 02:46:03.171375 17585 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:03.171422 17585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:03.171545 17585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:03.172595 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "8699ca2f05014379a2882ddc0ac3af2d"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "a678710f7788a28b5502dc43ffa67a44"
server_key_iv: "67fdfbedef011792513a156bc7943c6c"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.172911 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8699ca2f05014379a2882ddc0ac3af2d"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "a678710f7788a28b5502dc43ffa67a44"
server_key_iv: "67fdfbedef011792513a156bc7943c6c"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.174125 17585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:03.174861 17600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.175030 17585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:46:03.175109 17585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "8699ca2f05014379a2882ddc0ac3af2d"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "a678710f7788a28b5502dc43ffa67a44"
server_key_iv: "67fdfbedef011792513a156bc7943c6c"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.175165 17585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:03.201647 17585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:03.203699 17585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:03.203840 17585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:03.204056 17585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:03.204346 17585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:03.204409 17585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.204440 17585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:03.204464 17585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.210083 17585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41357
I20260927 02:46:03.210137 17713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41357 every 8 connection(s)
I20260927 02:46:03.210443 17585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:03.212565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 17585
I20260927 02:46:03.212690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:46:03.212893 392 external_mini_cluster.cc:1428] Setting key 8c525b255da288a17f28f669d58c506e
Sep 27 02:46:03 dist-test-slave-44kr krb5kdc[17344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:03.221159 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.211133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44311 (local address 127.0.98.62:46297)
0927 02:46:03.211193 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.211195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.211381 (+ 186us) server_negotiation.cc:407] Connection header received
0927 02:46:03.212367 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.212369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.212415 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.212485 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:03.213311 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.213950 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.215473 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.215797 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.215879 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.218155 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:03.218157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:03.218157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:03.218164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:03.219516 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:03.220012 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:03.220014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:03.220014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:03.220113 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:03.220443 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:03.220444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:03.220445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:03.220569 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:03.220631 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.220978 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.221057 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:46:03.221378 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.211104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46297 (local address 127.0.98.2:44311)
0927 02:46:03.211259 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.211303 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:03.212219 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:03.212594 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:03.212600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:03.212983 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:03.213242 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.213247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.214061 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.214063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.215343 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.215346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.215880 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.215881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.215996 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.216586 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:03.216596 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:03.218029 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:03.219598 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:03.219607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:03.219609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:03.219921 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:03.220183 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:03.220184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:03.220184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:03.220349 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:03.220639 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:03.220646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:03.220828 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 02:46:03.220955 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 02:46:03.221138 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:46:03.221922 17714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46297
I20260927 02:46:03.222090 17714 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:03.222283 17714 heartbeater.cc:511] Master 127.0.98.62:46297 requested a full tablet report, sending...
I20260927 02:46:03.222702 17389 ts_manager.cc:194] Registered new tserver with Master: 8699ca2f05014379a2882ddc0ac3af2d (127.0.98.2:41357)
I20260927 02:46:03.223071 17389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:46:03.240490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46297
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:03.320897 17719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:03.321106 17719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:03.321130 17719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:03.322253 17719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:03.322291 17719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:03.322316 17719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:03.323855 17719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:03.324060 17719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:03.324251 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:03.329460 17725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:03.329463 17727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:03.329460 17724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:03.329712 17719 server_base.cc:1034] running on GCE node
I20260927 02:46:03.329934 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:03.330211 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:03.331377 17719 hybrid_clock.cc:648] HybridClock initialized: now 1790477163331365 us; error 34 us; skew 500 ppm
Sep 27 02:46:03 dist-test-slave-44kr krb5kdc[17344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:03.333945 17719 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:46:03.334393 17719 webserver.cc:460] Webserver started at http://127.0.98.3:32989/ using document root <none> and password file <none>
I20260927 02:46:03.334666 17719 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:03.334734 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:03.334843 17719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:03.335996 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "47843877923a40a6a81359a9b4cb298c"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "b32a682f7abece0e34dde6b1bc02e72e"
server_key_iv: "f138685d5f318c0b954ee6467f88b94f"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.336419 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47843877923a40a6a81359a9b4cb298c"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "b32a682f7abece0e34dde6b1bc02e72e"
server_key_iv: "f138685d5f318c0b954ee6467f88b94f"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.337741 17719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:03.338438 17734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.338599 17719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.338711 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "47843877923a40a6a81359a9b4cb298c"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "b32a682f7abece0e34dde6b1bc02e72e"
server_key_iv: "f138685d5f318c0b954ee6467f88b94f"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.338781 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:03.356021 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:03.358187 17719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:03.358304 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:03.358541 17719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:03.358855 17719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:03.358891 17719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.358918 17719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:03.358937 17719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.364378 17719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45711
I20260927 02:46:03.364485 17847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45711 every 8 connection(s)
I20260927 02:46:03.364768 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:03.365206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 17719
I20260927 02:46:03.365299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:46:03.365481 392 external_mini_cluster.cc:1428] Setting key 990042055094e4241ef7cc9b9628cd04
Sep 27 02:46:03 dist-test-slave-44kr krb5kdc[17344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:03.374718 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.365506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55507 (local address 127.0.98.62:46297)
0927 02:46:03.365556 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.365558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.365776 (+ 218us) server_negotiation.cc:407] Connection header received
0927 02:46:03.366846 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.366847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.366857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.366898 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:03.367482 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.367980 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.369199 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.369388 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.369446 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.371825 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:03.371827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:03.371827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:03.371834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:03.373222 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:03.373656 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:03.373658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:03.373658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:03.373719 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:03.374066 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:03.374068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:03.374068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:03.374194 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:03.374256 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.374564 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.374634 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:46:03.374897 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.365473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46297 (local address 127.0.98.3:55507)
0927 02:46:03.365651 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.365694 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:03.366764 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:03.366961 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:03.366965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:03.367215 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:03.367402 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.367406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.368092 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.368093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.369108 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.369112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.369485 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.369486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.369608 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.370153 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:03.370163 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:03.371710 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:03.373298 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:03.373305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:03.373307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:03.373553 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:03.373825 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:03.373826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:03.373826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:03.373948 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:03.374269 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:03.374274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:03.374416 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 02:46:03.374510 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:46:03.374661 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260927 02:46:03.375447 17848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46297
I20260927 02:46:03.375546 17848 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:03.375718 17848 heartbeater.cc:511] Master 127.0.98.62:46297 requested a full tablet report, sending...
I20260927 02:46:03.376161 17389 ts_manager.cc:194] Registered new tserver with Master: 47843877923a40a6a81359a9b4cb298c (127.0.98.3:45711)
I20260927 02:46:03.376665 17389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55507
I20260927 02:46:03.377214 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:03.384931 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.378531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.0.98.62:46297)
0927 02:46:03.378597 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.378599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.378646 (+ 47us) server_negotiation.cc:407] Connection header received
0927 02:46:03.378695 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.378695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.378704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.378766 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:03.379310 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.379823 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.380688 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.380878 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.380939 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.382235 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:03.382236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:03.382236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:03.382242 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:03.383599 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:03.383938 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:03.383940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:03.383940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:03.383999 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:03.384289 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:03.384290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:03.384291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:03.384525 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:03.384606 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.384787 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.384862 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:46:03.386618 17389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:03.388144 17389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:03.394465 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.390495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41357 (local address 127.0.0.1:51606)
0927 02:46:03.390732 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.390742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:03.390796 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:03.391135 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:03.391135 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:03.391138 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:03.391183 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.391184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.392452 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.392453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.393522 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.393524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.394157 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.394158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.394232 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.394246 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:46:03.394311 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:46:03.394363 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":27,"thread_start_us":75,"threads_started":1}
I20260927 02:46:03.394532 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.390435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.0.98.1:34779)
0927 02:46:03.390909 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.390914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.390923 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:46:03.390953 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.390955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.390966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.391034 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:03.391216 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.391996 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.393472 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.394199 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.394280 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.394337 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.394399 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.394436 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":85,"threads_started":1}
I20260927 02:46:03.394800 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.390729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51606 (local address 127.0.98.2:41357)
0927 02:46:03.390881 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.390885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.390894 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:46:03.390945 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.390947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.390963 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.391041 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:03.391257 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.392304 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.393596 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.394084 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.394147 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.394207 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.394685 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.394727 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260927 02:46:03.394824 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.390482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.0.98.3:45711)
0927 02:46:03.391178 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.391182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.391192 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:46:03.391225 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.391226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.391241 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.391307 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:03.391510 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.392549 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.393623 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.394288 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.394340 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.394387 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.394736 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.394770 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":86,"threads_started":1}
I20260927 02:46:03.394520 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.390292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34779 (local address 127.0.0.1:55448)
0927 02:46:03.390506 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.390549 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:03.390615 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:03.391053 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:03.391056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:03.391059 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:03.391136 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.391138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.392100 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.392101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.393386 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.393388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.394280 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.394281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.394334 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.394341 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:46:03.394385 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:46:03.394417 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
I20260927 02:46:03.394676 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.390305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45711 (local address 127.0.0.1:34870)
0927 02:46:03.390608 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.390618 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:03.390657 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:03.391382 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:03.391383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:03.391385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:03.391426 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.391427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.392632 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.392633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.393536 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.393538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.394420 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.394421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.394494 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.394498 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:03.394549 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:46:03.394587 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":80,"threads_started":1}
I20260927 02:46:03.395718 17648 tablet_service.cc:1467] Processing CreateTablet for tablet 35ca745068504d6d8e93a3f92c6b3320 (DEFAULT_TABLE table=test-table [id=2b33c66f71114b7fbeb642ac2f8e5163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:03.396000 17648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ca745068504d6d8e93a3f92c6b3320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:03.395921 17782 tablet_service.cc:1467] Processing CreateTablet for tablet 35ca745068504d6d8e93a3f92c6b3320 (DEFAULT_TABLE table=test-table [id=2b33c66f71114b7fbeb642ac2f8e5163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:03.396221 17782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ca745068504d6d8e93a3f92c6b3320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:03.396801 17514 tablet_service.cc:1467] Processing CreateTablet for tablet 35ca745068504d6d8e93a3f92c6b3320 (DEFAULT_TABLE table=test-table [id=2b33c66f71114b7fbeb642ac2f8e5163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:03.397099 17514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ca745068504d6d8e93a3f92c6b3320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:03.398790 17865 tablet_bootstrap.cc:492] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: Bootstrap starting.
I20260927 02:46:03.399400 17865 tablet_bootstrap.cc:654] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:03.399407 17866 tablet_bootstrap.cc:492] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: Bootstrap starting.
I20260927 02:46:03.399919 17865 log.cc:824] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:03.400019 17866 tablet_bootstrap.cc:654] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:03.400046 17867 tablet_bootstrap.cc:492] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9: Bootstrap starting.
I20260927 02:46:03.400519 17866 log.cc:824] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:03.400664 17867 tablet_bootstrap.cc:654] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:03.401160 17867 log.cc:824] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:03.401178 17865 tablet_bootstrap.cc:492] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: No bootstrap required, opened a new log
I20260927 02:46:03.401186 17866 tablet_bootstrap.cc:492] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: No bootstrap required, opened a new log
I20260927 02:46:03.401235 17866 ts_tablet_manager.cc:1397] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:46:03.401237 17865 ts_tablet_manager.cc:1397] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:03.401758 17867 tablet_bootstrap.cc:492] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9: No bootstrap required, opened a new log
I20260927 02:46:03.401818 17867 ts_tablet_manager.cc:1397] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:46:03.403154 17866 raft_consensus.cc:348] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } }
I20260927 02:46:03.403154 17865 raft_consensus.cc:348] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } }
I20260927 02:46:03.403183 17867 raft_consensus.cc:348] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } }
I20260927 02:46:03.403331 17865 raft_consensus.cc:374] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:03.403344 17867 raft_consensus.cc:374] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:03.403373 17865 raft_consensus.cc:729] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8699ca2f05014379a2882ddc0ac3af2d, State: Initialized, Role: FOLLOWER
I20260927 02:46:03.403380 17867 raft_consensus.cc:729] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4da3b7c82640d286e93865348834f9, State: Initialized, Role: FOLLOWER
I20260927 02:46:03.403445 17866 raft_consensus.cc:374] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:03.403486 17866 raft_consensus.cc:729] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47843877923a40a6a81359a9b4cb298c, State: Initialized, Role: FOLLOWER
I20260927 02:46:03.403482 17867 consensus_queue.cc:260] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } }
I20260927 02:46:03.403497 17865 consensus_queue.cc:260] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } }
I20260927 02:46:03.403578 17866 consensus_queue.cc:260] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } }
I20260927 02:46:03.403748 17580 heartbeater.cc:503] Master 127.0.98.62:46297 was elected leader, sending a full tablet report...
I20260927 02:46:03.403769 17865 ts_tablet_manager.cc:1428] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:46:03.403793 17866 ts_tablet_manager.cc:1428] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:46:03.403895 17848 heartbeater.cc:503] Master 127.0.98.62:46297 was elected leader, sending a full tablet report...
I20260927 02:46:03.403898 17714 heartbeater.cc:503] Master 127.0.98.62:46297 was elected leader, sending a full tablet report...
I20260927 02:46:03.404054 17867 ts_tablet_manager.cc:1428] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260927 02:46:03.461011 17715 tablet.cc:2369] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:46:03.543625 17581 tablet.cc:2369] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:03.600783 17873 raft_consensus.cc:482] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:03.600910 17873 raft_consensus.cc:504] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } }
I20260927 02:46:03.601321 17873 leader_election.cc:288] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b4da3b7c82640d286e93865348834f9 (127.0.98.1:34779), 8699ca2f05014379a2882ddc0ac3af2d (127.0.98.2:41357)
I20260927 02:46:03.603991 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.601529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34779 (local address 127.0.98.3:47807)
0927 02:46:03.601588 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.601600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:03.601644 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:03.601797 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:03.601798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:03.601800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:03.601864 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.601866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.602461 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.602462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.603237 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.603240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.603753 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.603754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.603830 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.603833 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:03.603877 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:46:03.603914 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":23}
I20260927 02:46:03.603989 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.601588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47807 (local address 127.0.98.1:34779)
0927 02:46:03.601628 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.601629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.601639 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:46:03.601706 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.601707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.601715 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.601780 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:03.601923 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.602389 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.603321 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.603688 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.603746 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.603756 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.603889 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.603924 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:46:03.604108 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.601568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36007 (local address 127.0.98.2:41357)
0927 02:46:03.601612 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.601614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.601814 (+ 200us) server_negotiation.cc:407] Connection header received
0927 02:46:03.601847 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.601848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.601861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.601905 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:03.602041 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.602498 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.603416 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.603796 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.603855 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.603864 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.603996 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.604035 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:46:03.604120 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.601529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41357 (local address 127.0.98.3:36007)
0927 02:46:03.601750 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.601758 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:03.601801 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:03.601917 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:03.601918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:03.601919 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:03.601986 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.601987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.602574 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.602575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.603336 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.603338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.603866 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.603867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.603939 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.603943 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:03.603985 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:46:03.604022 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":76,"thread_start_us":106,"threads_started":1}
I20260927 02:46:03.604529 17534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ca745068504d6d8e93a3f92c6b3320" candidate_uuid: "47843877923a40a6a81359a9b4cb298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4da3b7c82640d286e93865348834f9" is_pre_election: true
I20260927 02:46:03.604526 17668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ca745068504d6d8e93a3f92c6b3320" candidate_uuid: "47843877923a40a6a81359a9b4cb298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8699ca2f05014379a2882ddc0ac3af2d" is_pre_election: true
I20260927 02:46:03.604688 17534 raft_consensus.cc:2454] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47843877923a40a6a81359a9b4cb298c in term 0.
I20260927 02:46:03.604688 17668 raft_consensus.cc:2454] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47843877923a40a6a81359a9b4cb298c in term 0.
I20260927 02:46:03.604912 17735 leader_election.cc:302] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47843877923a40a6a81359a9b4cb298c, 8699ca2f05014379a2882ddc0ac3af2d; no voters:
I20260927 02:46:03.605039 17873 raft_consensus.cc:2781] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:03.605087 17873 raft_consensus.cc:482] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:03.605105 17873 raft_consensus.cc:3037] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:03.605762 17873 raft_consensus.cc:504] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } }
I20260927 02:46:03.605881 17873 leader_election.cc:288] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [CANDIDATE]: Term 1 election: Requested vote from peers 6b4da3b7c82640d286e93865348834f9 (127.0.98.1:34779), 8699ca2f05014379a2882ddc0ac3af2d (127.0.98.2:41357)
I20260927 02:46:03.606045 17534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ca745068504d6d8e93a3f92c6b3320" candidate_uuid: "47843877923a40a6a81359a9b4cb298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4da3b7c82640d286e93865348834f9"
I20260927 02:46:03.606047 17668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ca745068504d6d8e93a3f92c6b3320" candidate_uuid: "47843877923a40a6a81359a9b4cb298c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8699ca2f05014379a2882ddc0ac3af2d"
I20260927 02:46:03.606123 17668 raft_consensus.cc:3037] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:03.606123 17534 raft_consensus.cc:3037] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:03.606947 17668 raft_consensus.cc:2454] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47843877923a40a6a81359a9b4cb298c in term 1.
I20260927 02:46:03.606947 17534 raft_consensus.cc:2454] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47843877923a40a6a81359a9b4cb298c in term 1.
I20260927 02:46:03.607132 17735 leader_election.cc:302] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47843877923a40a6a81359a9b4cb298c, 8699ca2f05014379a2882ddc0ac3af2d; no voters:
I20260927 02:46:03.607228 17873 raft_consensus.cc:2781] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:03.607362 17873 raft_consensus.cc:686] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 1 LEADER]: Becoming Leader. State: Replica: 47843877923a40a6a81359a9b4cb298c, State: Running, Role: LEADER
I20260927 02:46:03.607442 17873 consensus_queue.cc:237] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } }
I20260927 02:46:03.608354 17389 catalog_manager.cc:5495] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c reported cstate change: term changed from 0 to 1, leader changed from <none> to 47843877923a40a6a81359a9b4cb298c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47843877923a40a6a81359a9b4cb298c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47843877923a40a6a81359a9b4cb298c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 } health_report { overall_health: UNKNOWN } } }
W20260927 02:46:03.615329 17849 tablet.cc:2369] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:03.626189 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.623788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.0.98.3:45711)
0927 02:46:03.623834 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.623836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.623845 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:46:03.623922 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.623923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.623944 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.624010 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:03.624140 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.624640 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.625420 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.625635 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.625689 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.625844 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:03.625952 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:03.626025 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.626062 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.626084 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:46:03.627949 17534 raft_consensus.cc:1261] T 35ca745068504d6d8e93a3f92c6b3320 P 6b4da3b7c82640d286e93865348834f9 [term 1 FOLLOWER]: Refusing update from remote peer 47843877923a40a6a81359a9b4cb298c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:03.628042 17668 raft_consensus.cc:1261] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [term 1 FOLLOWER]: Refusing update from remote peer 47843877923a40a6a81359a9b4cb298c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:03.628335 17875 consensus_queue.cc:1035] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4da3b7c82640d286e93865348834f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:03.628515 17873 consensus_queue.cc:1035] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8699ca2f05014379a2882ddc0ac3af2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:03.631036 17879 mvcc.cc:205] Tried to move back new op lower bound from 7333794462217383936 to 7333794462136836096. Current Snapshot: MvccSnapshot[applied={T|T < 7333794462217383936}]
I20260927 02:46:03.631791 17881 mvcc.cc:205] Tried to move back new op lower bound from 7333794462217383936 to 7333794462136836096. Current Snapshot: MvccSnapshot[applied={T|T < 7333794462217383936}]
I20260927 02:46:03.633267 17389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:03.633368 17389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:03.634083 17389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a09b6abfe814649a06e59aced323edf: Sending DeleteTablet for 3 replicas of tablet 35ca745068504d6d8e93a3f92c6b3320
I20260927 02:46:03.634461 17782 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ca745068504d6d8e93a3f92c6b3320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34870
I20260927 02:46:03.634474 17648 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ca745068504d6d8e93a3f92c6b3320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51606
I20260927 02:46:03.634716 17514 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ca745068504d6d8e93a3f92c6b3320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55448
I20260927 02:46:03.634783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 17451
I20260927 02:46:03.635486 17889 tablet_replica.cc:330] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: stopping tablet replica
I20260927 02:46:03.635614 17889 raft_consensus.cc:2229] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:03.635726 17889 raft_consensus.cc:2258] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:03.635885 17888 tablet_replica.cc:330] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: stopping tablet replica
I20260927 02:46:03.635979 17888 raft_consensus.cc:2229] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:46:03.636076 17888 raft_consensus.cc:2258] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:03.636672 17889 ts_tablet_manager.cc:1905] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:03.638741 17889 ts_tablet_manager.cc:1918] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:03.638870 17889 log.cc:1196] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/35ca745068504d6d8e93a3f92c6b3320
I20260927 02:46:03.639257 17889 ts_tablet_manager.cc:1939] T 35ca745068504d6d8e93a3f92c6b3320 P 8699ca2f05014379a2882ddc0ac3af2d: Deleting consensus metadata
I20260927 02:46:03.639794 17373 catalog_manager.cc:4841] TS 8699ca2f05014379a2882ddc0ac3af2d (127.0.98.2:41357): tablet 35ca745068504d6d8e93a3f92c6b3320 (table test-table [id=2b33c66f71114b7fbeb642ac2f8e5163]) successfully deleted
W20260927 02:46:03.640802 17374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34779 (error 108)
I20260927 02:46:03.641256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 17585
I20260927 02:46:03.641424 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.641265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34779 (local address 127.0.0.1:55452)
0927 02:46:03.641361 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.641391 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 02:46:03.642093 17888 ts_tablet_manager.cc:1905] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:03.643659 17888 ts_tablet_manager.cc:1918] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:03.643730 17888 log.cc:1196] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/35ca745068504d6d8e93a3f92c6b3320
I20260927 02:46:03.643971 17888 ts_tablet_manager.cc:1939] T 35ca745068504d6d8e93a3f92c6b3320 P 47843877923a40a6a81359a9b4cb298c: Deleting consensus metadata
W20260927 02:46:03.644155 17374 catalog_manager.cc:4584] TS 6b4da3b7c82640d286e93865348834f9 (127.0.98.1:34779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35ca745068504d6d8e93a3f92c6b3320: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34779: connect: Connection refused (error 111)
I20260927 02:46:03.644565 17373 catalog_manager.cc:4841] TS 47843877923a40a6a81359a9b4cb298c (127.0.98.3:45711): tablet 35ca745068504d6d8e93a3f92c6b3320 (table test-table [id=2b33c66f71114b7fbeb642ac2f8e5163]) successfully deleted
I20260927 02:46:03.648108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 17719
I20260927 02:46:03.652820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 17357
2026-09-27T02:46:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2042 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T02:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:03Z Disabled control of system clock
I20260927 02:46:03.680902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33827
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:46:03.760346 17895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:03.760598 17895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:03.760619 17895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:03.761752 17895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:03.761786 17895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:03.761797 17895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:03.761808 17895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:03.761818 17895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:03.763110 17895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:03.763309 17895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:03.763547 17895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:46:03.768553 17895 server_base.cc:1034] running on GCE node
W20260927 02:46:03.768543 17900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:03.768517 17903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:03.768496 17901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:03.769076 17895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:03.769363 17895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:03.770534 17895 hybrid_clock.cc:648] HybridClock initialized: now 1790477163770526 us; error 43 us; skew 500 ppm
I20260927 02:46:03.771850 17895 webserver.cc:460] Webserver started at http://127.0.98.62:36409/ using document root <none> and password file <none>
I20260927 02:46:03.772063 17895 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:03.772166 17895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:03.772287 17895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:03.773376 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "7fbf90d2349b4aa09b16add1daf78a1b"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "be56fd4f572d202130314fdd8bca0a7a"
server_key_iv: "dbea33e9c9014edf90bff02f662399b9"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.773705 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "7fbf90d2349b4aa09b16add1daf78a1b"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "be56fd4f572d202130314fdd8bca0a7a"
server_key_iv: "dbea33e9c9014edf90bff02f662399b9"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.774948 17895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:46:03.775723 17909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.775938 17895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:03.776026 17895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "7fbf90d2349b4aa09b16add1daf78a1b"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "be56fd4f572d202130314fdd8bca0a7a"
server_key_iv: "dbea33e9c9014edf90bff02f662399b9"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.776095 17895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:03.789274 17895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:03.789592 17895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:03.789724 17895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:03.793584 17895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33827
I20260927 02:46:03.793673 17961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33827 every 8 connection(s)
I20260927 02:46:03.793943 17895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:03.794855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 17895
I20260927 02:46:03.794989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:03.795017 17962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:03.795229 392 external_mini_cluster.cc:1428] Setting key 947cd7657d070a0b1a1b65f7a1e02050
I20260927 02:46:03.798014 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b: Bootstrap starting.
I20260927 02:46:03.798575 17962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:03.798882 17962 log.cc:824] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:03.799661 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b: No bootstrap required, opened a new log
I20260927 02:46:03.800873 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.795788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.0.98.62:33827)
0927 02:46:03.795969 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.795976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.795999 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:46:03.796562 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.796569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.796634 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.796760 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:03.796983 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.797850 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.798790 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.799360 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.799488 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.800173 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:03.800177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:03.800177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:03.800373 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:03.800447 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:03.800462 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:03.800532 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.800598 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.800675 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260927 02:46:03.801391 17962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbf90d2349b4aa09b16add1daf78a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260927 02:46:03.801535 17962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:03.801568 17962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbf90d2349b4aa09b16add1daf78a1b, State: Initialized, Role: FOLLOWER
I20260927 02:46:03.801692 17962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbf90d2349b4aa09b16add1daf78a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260927 02:46:03.801774 17962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:03.801805 17962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:03.801851 17962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:03.802353 17962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbf90d2349b4aa09b16add1daf78a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260927 02:46:03.802446 17962 leader_election.cc:302] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fbf90d2349b4aa09b16add1daf78a1b; no voters:
I20260927 02:46:03.802620 17962 leader_election.cc:288] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:03.802678 17967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:03.802837 17967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [term 1 LEADER]: Becoming Leader. State: Replica: 7fbf90d2349b4aa09b16add1daf78a1b, State: Running, Role: LEADER
I20260927 02:46:03.802856 17962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:03.802932 17967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbf90d2349b4aa09b16add1daf78a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260927 02:46:03.803352 17968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fbf90d2349b4aa09b16add1daf78a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbf90d2349b4aa09b16add1daf78a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } } }
I20260927 02:46:03.803367 17969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fbf90d2349b4aa09b16add1daf78a1b. Latest consensus state: current_term: 1 leader_uuid: "7fbf90d2349b4aa09b16add1daf78a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbf90d2349b4aa09b16add1daf78a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } } }
I20260927 02:46:03.803448 17968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:03.803454 17969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:03.803634 17976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:03.804121 17976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:03.805564 17976 catalog_manager.cc:1347] Generated new cluster ID: 4c16073632994103ae717505d072fed2
I20260927 02:46:03.805622 17976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:03.819449 17976 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:03.819978 17976 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:03.827665 17976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b: Generated new TSK 0
I20260927 02:46:03.827838 17976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:03.835314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--builtin_ntp_servers=127.0.98.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:46:03.915192 17986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:03.915405 17986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:03.915431 17986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:03.916626 17986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:03.916666 17986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:03.916697 17986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:03.918028 17986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:03.918252 17986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:03.918522 17986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:03.923259 17994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:03.923280 17992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:03.923468 17986 server_base.cc:1034] running on GCE node
W20260927 02:46:03.923270 17991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:03.923785 17986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:03.924067 17986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:03.925225 17986 hybrid_clock.cc:648] HybridClock initialized: now 1790477163925210 us; error 31 us; skew 500 ppm
I20260927 02:46:03.926602 17986 webserver.cc:460] Webserver started at http://127.0.98.1:37941/ using document root <none> and password file <none>
I20260927 02:46:03.926854 17986 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:03.926918 17986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:03.927030 17986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:03.928036 17986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "e9f48375b22844d5a413fe337bf157f3"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "426ce6f13ed9bf3bd74c7ab2d32aadf6"
server_key_iv: "2c70843c3b5895406d44c4e3946c4c76"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.928309 17986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e9f48375b22844d5a413fe337bf157f3"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "426ce6f13ed9bf3bd74c7ab2d32aadf6"
server_key_iv: "2c70843c3b5895406d44c4e3946c4c76"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.929497 17986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:03.930265 18000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.930415 17986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:46:03.930498 17986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "e9f48375b22844d5a413fe337bf157f3"
format_stamp: "Formatted at 2026-09-27 02:46:03 on dist-test-slave-44kr"
server_key: "426ce6f13ed9bf3bd74c7ab2d32aadf6"
server_key_iv: "2c70843c3b5895406d44c4e3946c4c76"
server_key_version: "encryptionkey@0"
I20260927 02:46:03.930568 17986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:03.956075 17986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:03.956450 17986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:03.956589 17986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:03.956836 17986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:03.957177 17986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:03.957211 17986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.957239 17986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:03.957265 17986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:03.962589 17986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41691
I20260927 02:46:03.962642 18113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41691 every 8 connection(s)
I20260927 02:46:03.962954 17986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:03.967730 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.963658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45875 (local address 127.0.98.62:33827)
0927 02:46:03.963702 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:03.963704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:03.963933 (+ 229us) server_negotiation.cc:407] Connection header received
0927 02:46:03.964416 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:03.964418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:03.964434 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:03.964490 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:03.965146 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.965653 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.966571 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.966761 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.966810 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.967500 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:03.967501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:03.967501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:03.967505 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:03.967519 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:03.967522 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:03.967559 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:03.967637 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:46:03.967669 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:46:03.967873 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:03.963652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33827 (local address 127.0.98.1:45875)
0927 02:46:03.963816 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:03.963850 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:03.964317 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:03.964546 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:03.964550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:03.964559 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:03.965065 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.965069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.965721 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.965721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.966501 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:03.966504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:03.966825 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:03.966825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:03.966932 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:03.967383 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:46:03.967383 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:03.967410 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:46:03.967431 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:03.967560 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:03.967562 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:03.967564 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:46:03.967623 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:46:03.967700 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260927 02:46:03.968374 18114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33827
I20260927 02:46:03.968473 18114 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:03.968654 18114 heartbeater.cc:511] Master 127.0.98.62:33827 requested a full tablet report, sending...
I20260927 02:46:03.969129 17926 ts_manager.cc:194] Registered new tserver with Master: e9f48375b22844d5a413fe337bf157f3 (127.0.98.1:41691)
I20260927 02:46:03.969738 17926 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45875
I20260927 02:46:03.970311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 17986
I20260927 02:46:03.970403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:46:03.970628 392 external_mini_cluster.cc:1428] Setting key 6846ccdb14f39511fd665098f90087dc
I20260927 02:46:03.971593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--builtin_ntp_servers=127.0.98.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:46:04.051277 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:04.051484 18117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:04.051514 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:04.052704 18117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:04.052747 18117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:04.052779 18117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:04.054145 18117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:04.054386 18117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:04.054697 18117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:04.059434 18123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:04.059442 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:04.059440 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:04.059577 18117 server_base.cc:1034] running on GCE node
I20260927 02:46:04.059816 18117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:04.060096 18117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:04.061252 18117 hybrid_clock.cc:648] HybridClock initialized: now 1790477164061234 us; error 29 us; skew 500 ppm
I20260927 02:46:04.062436 18117 webserver.cc:460] Webserver started at http://127.0.98.2:44883/ using document root <none> and password file <none>
I20260927 02:46:04.062654 18117 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:04.062704 18117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:04.062829 18117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:04.063926 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "780777da5e5a48fc929856eef35cee6a"
format_stamp: "Formatted at 2026-09-27 02:46:04 on dist-test-slave-44kr"
server_key: "43b90d805ae50987a16ad6c055b7bf09"
server_key_iv: "10ddd74f8eb0bbac8ae82f87102337cd"
server_key_version: "encryptionkey@0"
I20260927 02:46:04.064265 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "780777da5e5a48fc929856eef35cee6a"
format_stamp: "Formatted at 2026-09-27 02:46:04 on dist-test-slave-44kr"
server_key: "43b90d805ae50987a16ad6c055b7bf09"
server_key_iv: "10ddd74f8eb0bbac8ae82f87102337cd"
server_key_version: "encryptionkey@0"
I20260927 02:46:04.065537 18117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:04.066360 18131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:04.066546 18117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:04.066627 18117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "780777da5e5a48fc929856eef35cee6a"
format_stamp: "Formatted at 2026-09-27 02:46:04 on dist-test-slave-44kr"
server_key: "43b90d805ae50987a16ad6c055b7bf09"
server_key_iv: "10ddd74f8eb0bbac8ae82f87102337cd"
server_key_version: "encryptionkey@0"
I20260927 02:46:04.066695 18117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:04.084076 18117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:04.084447 18117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:04.084586 18117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:04.084812 18117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:04.085150 18117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:04.085187 18117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:04.085216 18117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:04.085237 18117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:04.090756 18117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38239
I20260927 02:46:04.090802 18244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38239 every 8 connection(s)
I20260927 02:46:04.091140 18117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:04.096510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 18117
I20260927 02:46:04.096578 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.091871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45031 (local address 127.0.98.62:33827)
0927 02:46:04.091927 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:04.091929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:04.092249 (+ 320us) server_negotiation.cc:407] Connection header received
0927 02:46:04.092772 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:04.092773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:04.092784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:04.092847 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:04.093578 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.094091 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.095242 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.095437 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.095493 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.096294 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:04.096295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:04.096296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:04.096300 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:04.096315 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:04.096318 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:04.096357 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:04.096481 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 02:46:04.096512 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:46:04.096633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:46:04.096848 392 external_mini_cluster.cc:1428] Setting key 699327aa70cf23ad8b40fcea7f9d9523
I20260927 02:46:04.096884 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.091862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33827 (local address 127.0.98.2:45031)
0927 02:46:04.092107 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:04.092143 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:04.092671 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:04.092907 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:04.092911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:04.092920 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:04.093430 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.093435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.094213 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.094214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.095145 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.095148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.095520 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.095521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.095643 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.096152 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:46:04.096153 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:04.096185 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:46:04.096209 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:04.096374 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:04.096379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:04.096403 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:46:04.096477 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:46:04.096587 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260927 02:46:04.097411 18245 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33827
I20260927 02:46:04.097523 18245 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:04.097772 18245 heartbeater.cc:511] Master 127.0.98.62:33827 requested a full tablet report, sending...
I20260927 02:46:04.097993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--builtin_ntp_servers=127.0.98.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:46:04.098223 17925 ts_manager.cc:194] Registered new tserver with Master: 780777da5e5a48fc929856eef35cee6a (127.0.98.2:38239)
I20260927 02:46:04.098635 17925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45031
W20260927 02:46:04.178638 18248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:04.178864 18248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:04.178887 18248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:04.180111 18248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:04.180154 18248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:04.180178 18248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:04.181541 18248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:04.181766 18248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:04.182037 18248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:04.186808 18254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:04.186815 18253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:04.186878 18248 server_base.cc:1034] running on GCE node
W20260927 02:46:04.187090 18256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:04.187309 18248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:04.187529 18248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:04.188660 18248 hybrid_clock.cc:648] HybridClock initialized: now 1790477164188644 us; error 27 us; skew 500 ppm
I20260927 02:46:04.189797 18248 webserver.cc:460] Webserver started at http://127.0.98.3:39651/ using document root <none> and password file <none>
I20260927 02:46:04.190011 18248 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:04.190059 18248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:04.190215 18248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:04.191237 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "91bb8ba83af94bf7ae68df65a43f597a"
format_stamp: "Formatted at 2026-09-27 02:46:04 on dist-test-slave-44kr"
server_key: "b7a67480197a21062d2dfb1e8c1c09a3"
server_key_iv: "c385d28bb9b60c708cb6dd47b84f8831"
server_key_version: "encryptionkey@0"
I20260927 02:46:04.191562 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "91bb8ba83af94bf7ae68df65a43f597a"
format_stamp: "Formatted at 2026-09-27 02:46:04 on dist-test-slave-44kr"
server_key: "b7a67480197a21062d2dfb1e8c1c09a3"
server_key_iv: "c385d28bb9b60c708cb6dd47b84f8831"
server_key_version: "encryptionkey@0"
I20260927 02:46:04.192754 18248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:46:04.193424 18262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:04.193570 18248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:46:04.193637 18248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "91bb8ba83af94bf7ae68df65a43f597a"
format_stamp: "Formatted at 2026-09-27 02:46:04 on dist-test-slave-44kr"
server_key: "b7a67480197a21062d2dfb1e8c1c09a3"
server_key_iv: "c385d28bb9b60c708cb6dd47b84f8831"
server_key_version: "encryptionkey@0"
I20260927 02:46:04.193701 18248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:04.204531 18248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:04.204823 18248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:04.204942 18248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:04.205150 18248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:04.205456 18248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:04.205487 18248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:04.205518 18248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:04.205541 18248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:04.210793 18248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34737
I20260927 02:46:04.210839 18375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34737 every 8 connection(s)
I20260927 02:46:04.211162 18248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:04.212894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 18248
I20260927 02:46:04.213006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:46:04.213212 392 external_mini_cluster.cc:1428] Setting key 9d8c5eaa33500b2c0707d134a6362389
I20260927 02:46:04.216691 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.211864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48501 (local address 127.0.98.62:33827)
0927 02:46:04.211934 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:04.211936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:04.212173 (+ 237us) server_negotiation.cc:407] Connection header received
0927 02:46:04.212687 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:04.212688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:04.212697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:04.212754 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:04.213663 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.214389 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.215369 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.215553 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.215606 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.216346 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:04.216347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:04.216347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:04.216351 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:04.216365 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:04.216369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:04.216470 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:04.216596 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 02:46:04.216626 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:46:04.216861 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.211835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33827 (local address 127.0.98.3:48501)
0927 02:46:04.212031 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:04.212081 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:04.212600 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:04.212837 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:04.212841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:04.212858 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:04.213576 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.213583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.214474 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.214474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.215290 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.215292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.215619 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.215620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.215729 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.216210 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:46:04.216211 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:04.216242 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:46:04.216266 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:04.216480 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:04.216487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:04.216490 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:04.216561 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:46:04.216651 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260927 02:46:04.217334 18376 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33827
I20260927 02:46:04.217423 18376 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:04.217573 18376 heartbeater.cc:511] Master 127.0.98.62:33827 requested a full tablet report, sending...
I20260927 02:46:04.217979 17925 ts_manager.cc:194] Registered new tserver with Master: 91bb8ba83af94bf7ae68df65a43f597a (127.0.98.3:34737)
I20260927 02:46:04.218356 17925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48501
I20260927 02:46:04.224949 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:04.229749 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.226325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.0.98.62:33827)
0927 02:46:04.226401 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:04.226403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:04.226417 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:46:04.226573 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:04.226574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:04.226584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:04.226635 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:04.226810 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.227298 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.228421 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.228724 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.228790 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.229503 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:04.229504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:04.229504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:04.229508 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:04.229524 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:04.229528 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:04.229578 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:04.229643 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:46:04.229667 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:46:04.231436 17925 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:04.233542 17925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:04.239602 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.235660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34737 (local address 127.0.0.1:43606)
0927 02:46:04.236150 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:04.236156 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:04.236180 (+ 24us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:04.236432 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:04.236432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:04.236435 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:04.236482 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.236483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.237385 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.237386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.238484 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.238487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.239303 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.239304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.239377 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.239391 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:46:04.239459 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:46:04.239511 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":7,"spinlock_wait_cycles":1280,"thread_start_us":111,"threads_started":1}
I20260927 02:46:04.239603 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.235915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.0.98.3:34737)
0927 02:46:04.236261 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:04.236265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:04.236272 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:46:04.236305 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:04.236307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:04.236322 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:04.236563 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:04.236627 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.237310 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.238576 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.239230 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.239289 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.239383 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:04.239476 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 02:46:04.239511 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":84,"threads_started":1}
I20260927 02:46:04.239765 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.235660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41691 (local address 127.0.0.1:44564)
0927 02:46:04.235927 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:04.235970 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:04.236040 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:04.236289 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:04.236292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:04.236295 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:04.236363 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.236365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.237769 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.237770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.238626 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.238628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.239528 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.239529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.239588 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.239592 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:04.239639 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:46:04.239681 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":127,"threads_started":1}
I20260927 02:46:04.239921 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.235661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38239 (local address 127.0.0.1:54256)
0927 02:46:04.236018 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:04.236028 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:04.236086 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:04.236591 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:04.236591 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:04.236593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:04.236633 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.236634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.237870 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.237871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.238894 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.238896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.239749 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.239749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.239801 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.239804 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:04.239832 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:46:04.239866 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":65,"threads_started":1}
I20260927 02:46:04.240043 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.235846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.0.98.1:41691)
0927 02:46:04.236151 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:04.236156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:04.236166 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:46:04.236200 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:04.236202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:04.236213 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:04.236279 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:04.236491 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.237674 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.238704 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.239345 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.239411 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.239493 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:04.239917 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 02:46:04.239967 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":79,"threads_started":1}
I20260927 02:46:04.240298 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.235844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.0.98.2:38239)
0927 02:46:04.236433 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:04.236437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:04.236446 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:46:04.236485 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:04.236487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:04.236501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:04.236706 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:04.236768 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.237794 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.238975 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.239668 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.240069 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.240163 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:04.240187 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 02:46:04.240219 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":162,"threads_started":1}
I20260927 02:46:04.240777 18048 tablet_service.cc:1467] Processing CreateTablet for tablet d265f00d7dce49408807cabd134fa364 (DEFAULT_TABLE table=test-table [id=64c58766cf234ed4bc686d6684cdc6ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:04.240777 18310 tablet_service.cc:1467] Processing CreateTablet for tablet d265f00d7dce49408807cabd134fa364 (DEFAULT_TABLE table=test-table [id=64c58766cf234ed4bc686d6684cdc6ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:04.240878 18179 tablet_service.cc:1467] Processing CreateTablet for tablet d265f00d7dce49408807cabd134fa364 (DEFAULT_TABLE table=test-table [id=64c58766cf234ed4bc686d6684cdc6ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:04.241135 18310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d265f00d7dce49408807cabd134fa364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:04.241137 18048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d265f00d7dce49408807cabd134fa364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:04.241178 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d265f00d7dce49408807cabd134fa364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:04.243639 18390 tablet_bootstrap.cc:492] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: Bootstrap starting.
I20260927 02:46:04.244261 18390 tablet_bootstrap.cc:654] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:04.244354 18391 tablet_bootstrap.cc:492] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: Bootstrap starting.
I20260927 02:46:04.244912 18390 log.cc:824] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:04.245007 18391 tablet_bootstrap.cc:654] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:04.245296 18391 log.cc:824] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:04.245617 18392 tablet_bootstrap.cc:492] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3: Bootstrap starting.
I20260927 02:46:04.245738 18390 tablet_bootstrap.cc:492] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: No bootstrap required, opened a new log
I20260927 02:46:04.245800 18390 ts_tablet_manager.cc:1397] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:04.246233 18392 tablet_bootstrap.cc:654] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:04.246541 18392 log.cc:824] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:04.247076 18391 tablet_bootstrap.cc:492] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: No bootstrap required, opened a new log
I20260927 02:46:04.247146 18391 ts_tablet_manager.cc:1397] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 02:46:04.247347 18392 tablet_bootstrap.cc:492] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3: No bootstrap required, opened a new log
I20260927 02:46:04.247413 18392 ts_tablet_manager.cc:1397] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:04.248330 18390 raft_consensus.cc:348] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } }
I20260927 02:46:04.248553 18390 raft_consensus.cc:374] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:04.248600 18390 raft_consensus.cc:729] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780777da5e5a48fc929856eef35cee6a, State: Initialized, Role: FOLLOWER
I20260927 02:46:04.248744 18390 consensus_queue.cc:260] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } }
I20260927 02:46:04.249033 18390 ts_tablet_manager.cc:1428] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:04.249060 18245 heartbeater.cc:503] Master 127.0.98.62:33827 was elected leader, sending a full tablet report...
I20260927 02:46:04.249562 18391 raft_consensus.cc:348] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } }
I20260927 02:46:04.249720 18391 raft_consensus.cc:374] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:04.249754 18391 raft_consensus.cc:729] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bb8ba83af94bf7ae68df65a43f597a, State: Initialized, Role: FOLLOWER
I20260927 02:46:04.249665 18392 raft_consensus.cc:348] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } }
I20260927 02:46:04.249801 18392 raft_consensus.cc:374] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:04.249840 18392 raft_consensus.cc:729] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f48375b22844d5a413fe337bf157f3, State: Initialized, Role: FOLLOWER
I20260927 02:46:04.249886 18391 consensus_queue.cc:260] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } }
I20260927 02:46:04.249955 18392 consensus_queue.cc:260] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } }
I20260927 02:46:04.250173 18391 ts_tablet_manager.cc:1428] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 02:46:04.250229 18392 ts_tablet_manager.cc:1428] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:04.250342 18376 heartbeater.cc:503] Master 127.0.98.62:33827 was elected leader, sending a full tablet report...
I20260927 02:46:04.250370 18114 heartbeater.cc:503] Master 127.0.98.62:33827 was elected leader, sending a full tablet report...
W20260927 02:46:04.341732 18246 tablet.cc:2369] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:04.408589 18397 raft_consensus.cc:482] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:04.408727 18397 raft_consensus.cc:504] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } }
I20260927 02:46:04.409139 18397 leader_election.cc:288] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 780777da5e5a48fc929856eef35cee6a (127.0.98.2:38239), e9f48375b22844d5a413fe337bf157f3 (127.0.98.1:41691)
I20260927 02:46:04.412063 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.409489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42993 (local address 127.0.98.1:41691)
0927 02:46:04.409529 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:04.409530 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:04.409657 (+ 127us) server_negotiation.cc:407] Connection header received
0927 02:46:04.409689 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:04.409690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:04.409701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:04.409755 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:04.409950 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.410398 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.411369 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.411722 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.411784 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.411794 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:04.411957 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 02:46:04.411992 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:46:04.412068 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.409418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54795 (local address 127.0.98.2:38239)
0927 02:46:04.409465 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:04.409467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:04.409477 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:46:04.409518 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:04.409519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:04.409528 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:04.409594 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:04.409792 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.410411 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.411337 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.411706 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.411773 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.411785 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:04.411961 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:46:04.412000 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:46:04.412091 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.409375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41691 (local address 127.0.98.3:42993)
0927 02:46:04.409584 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:04.409594 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:04.409649 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:04.409792 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:04.409793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:04.409796 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:04.409869 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.409871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.410480 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.410480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.411277 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.411279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.411809 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.411810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.411880 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.411885 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:46:04.411944 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:46:04.411983 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260927 02:46:04.412060 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.409343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38239 (local address 127.0.98.3:54795)
0927 02:46:04.409384 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:04.409397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:04.409444 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:04.409603 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:04.409604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:04.409606 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:04.409674 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.409677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.410502 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.410503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.411250 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:04.411252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.411786 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.411787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:04.411856 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.411859 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:04.411897 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 02:46:04.411931 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 02:46:04.412547 18068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265f00d7dce49408807cabd134fa364" candidate_uuid: "91bb8ba83af94bf7ae68df65a43f597a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f48375b22844d5a413fe337bf157f3" is_pre_election: true
I20260927 02:46:04.412678 18199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265f00d7dce49408807cabd134fa364" candidate_uuid: "91bb8ba83af94bf7ae68df65a43f597a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780777da5e5a48fc929856eef35cee6a" is_pre_election: true
I20260927 02:46:04.412731 18068 raft_consensus.cc:2454] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bb8ba83af94bf7ae68df65a43f597a in term 0.
I20260927 02:46:04.412808 18199 raft_consensus.cc:2454] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bb8ba83af94bf7ae68df65a43f597a in term 0.
I20260927 02:46:04.412982 18265 leader_election.cc:302] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91bb8ba83af94bf7ae68df65a43f597a, e9f48375b22844d5a413fe337bf157f3; no voters:
I20260927 02:46:04.413137 18397 raft_consensus.cc:2781] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:04.413210 18397 raft_consensus.cc:482] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:04.413228 18397 raft_consensus.cc:3037] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:04.413928 18397 raft_consensus.cc:504] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } }
I20260927 02:46:04.414047 18397 leader_election.cc:288] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [CANDIDATE]: Term 1 election: Requested vote from peers 780777da5e5a48fc929856eef35cee6a (127.0.98.2:38239), e9f48375b22844d5a413fe337bf157f3 (127.0.98.1:41691)
I20260927 02:46:04.414247 18068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265f00d7dce49408807cabd134fa364" candidate_uuid: "91bb8ba83af94bf7ae68df65a43f597a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f48375b22844d5a413fe337bf157f3"
I20260927 02:46:04.414327 18068 raft_consensus.cc:3037] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:04.415004 18068 raft_consensus.cc:2454] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bb8ba83af94bf7ae68df65a43f597a in term 1.
I20260927 02:46:04.415104 18199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d265f00d7dce49408807cabd134fa364" candidate_uuid: "91bb8ba83af94bf7ae68df65a43f597a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780777da5e5a48fc929856eef35cee6a"
I20260927 02:46:04.415174 18265 leader_election.cc:302] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91bb8ba83af94bf7ae68df65a43f597a, e9f48375b22844d5a413fe337bf157f3; no voters:
I20260927 02:46:04.415186 18199 raft_consensus.cc:3037] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:04.415266 18397 raft_consensus.cc:2781] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:04.415402 18397 raft_consensus.cc:686] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 1 LEADER]: Becoming Leader. State: Replica: 91bb8ba83af94bf7ae68df65a43f597a, State: Running, Role: LEADER
I20260927 02:46:04.415575 18397 consensus_queue.cc:237] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } }
I20260927 02:46:04.416009 18199 raft_consensus.cc:2454] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bb8ba83af94bf7ae68df65a43f597a in term 1.
I20260927 02:46:04.416430 17925 catalog_manager.cc:5495] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a reported cstate change: term changed from 0 to 1, leader changed from <none> to 91bb8ba83af94bf7ae68df65a43f597a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "91bb8ba83af94bf7ae68df65a43f597a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91bb8ba83af94bf7ae68df65a43f597a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34737 } health_report { overall_health: HEALTHY } } }
I20260927 02:46:04.425309 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.422681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43610 (local address 127.0.98.3:34737)
0927 02:46:04.422733 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:04.422735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:04.422743 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:04.422775 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:04.422776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:04.422792 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:04.422864 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:04.423020 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.423522 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.424332 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:04.424598 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:04.424662 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:04.424871 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:04.425039 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:04.425115 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:04.425185 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:46:04.425212 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:46:04.427476 18199 raft_consensus.cc:1261] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [term 1 FOLLOWER]: Refusing update from remote peer 91bb8ba83af94bf7ae68df65a43f597a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:04.427476 18068 raft_consensus.cc:1261] T d265f00d7dce49408807cabd134fa364 P e9f48375b22844d5a413fe337bf157f3 [term 1 FOLLOWER]: Refusing update from remote peer 91bb8ba83af94bf7ae68df65a43f597a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:04.427848 18397 consensus_queue.cc:1035] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [LEADER]: Connected to new peer: Peer: permanent_uuid: "780777da5e5a48fc929856eef35cee6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:04.428088 18400 consensus_queue.cc:1035] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f48375b22844d5a413fe337bf157f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:04.431124 18405 mvcc.cc:205] Tried to move back new op lower bound from 7333794465491435520 to 7333794465446875136. Current Snapshot: MvccSnapshot[applied={T|T < 7333794465491435520}]
I20260927 02:46:04.430860 18404 mvcc.cc:205] Tried to move back new op lower bound from 7333794465491435520 to 7333794465446875136. Current Snapshot: MvccSnapshot[applied={T|T < 7333794465491435520}]
I20260927 02:46:04.432881 17925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:04.432960 17925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:04.433631 17925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fbf90d2349b4aa09b16add1daf78a1b: Sending DeleteTablet for 3 replicas of tablet d265f00d7dce49408807cabd134fa364
I20260927 02:46:04.433924 18179 tablet_service.cc:1514] Processing DeleteTablet for tablet d265f00d7dce49408807cabd134fa364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:04 UTC) from {username='slave'} at 127.0.0.1:54256
I20260927 02:46:04.433919 18310 tablet_service.cc:1514] Processing DeleteTablet for tablet d265f00d7dce49408807cabd134fa364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:04 UTC) from {username='slave'} at 127.0.0.1:43606
I20260927 02:46:04.434211 18413 tablet_replica.cc:330] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: stopping tablet replica
I20260927 02:46:04.434208 18048 tablet_service.cc:1514] Processing DeleteTablet for tablet d265f00d7dce49408807cabd134fa364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:04 UTC) from {username='slave'} at 127.0.0.1:44564
I20260927 02:46:04.434307 18413 raft_consensus.cc:2229] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:46:04.434396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 17986
I20260927 02:46:04.435806 18413 raft_consensus.cc:2258] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:04.435968 18414 tablet_replica.cc:330] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: stopping tablet replica
I20260927 02:46:04.436057 18414 raft_consensus.cc:2229] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:04.436156 18414 raft_consensus.cc:2258] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:04.436717 18414 ts_tablet_manager.cc:1905] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:04.438290 18414 ts_tablet_manager.cc:1918] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:04.438371 18414 log.cc:1196] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/d265f00d7dce49408807cabd134fa364
I20260927 02:46:04.438637 18414 ts_tablet_manager.cc:1939] T d265f00d7dce49408807cabd134fa364 P 780777da5e5a48fc929856eef35cee6a: Deleting consensus metadata
I20260927 02:46:04.439100 17913 catalog_manager.cc:4841] TS 780777da5e5a48fc929856eef35cee6a (127.0.98.2:38239): tablet d265f00d7dce49408807cabd134fa364 (table test-table [id=64c58766cf234ed4bc686d6684cdc6ad]) successfully deleted
I20260927 02:46:04.440626 18413 ts_tablet_manager.cc:1905] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:04.441979 18413 ts_tablet_manager.cc:1918] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:04.442058 18413 log.cc:1196] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/d265f00d7dce49408807cabd134fa364
I20260927 02:46:04.442301 18413 ts_tablet_manager.cc:1939] T d265f00d7dce49408807cabd134fa364 P 91bb8ba83af94bf7ae68df65a43f597a: Deleting consensus metadata
I20260927 02:46:04.442745 17911 catalog_manager.cc:4841] TS 91bb8ba83af94bf7ae68df65a43f597a (127.0.98.3:34737): tablet d265f00d7dce49408807cabd134fa364 (table test-table [id=64c58766cf234ed4bc686d6684cdc6ad]) successfully deleted
W20260927 02:46:04.443409 17912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41691 (error 108)
I20260927 02:46:04.443496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 18117
I20260927 02:46:04.445506 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:04.445380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41691 (local address 127.0.0.1:44570)
0927 02:46:04.445435 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:04.445476 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 02:46:04.445787 17912 catalog_manager.cc:4584] TS e9f48375b22844d5a413fe337bf157f3 (127.0.98.1:41691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d265f00d7dce49408807cabd134fa364: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41691: connect: Connection refused (error 111)
I20260927 02:46:04.449723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 18248
I20260927 02:46:04.454025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 17895
2026-09-27T02:46:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (799 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:04 dist-test-slave-44kr krb5kdc[18419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:04 dist-test-slave-44kr krb5kdc[18419](info): set up 2 sockets
Sep 27 02:46:04 dist-test-slave-44kr krb5kdc[18419](info): commencing operation
krb5kdc: starting...
W20260927 02:46:05.400024 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:05 dist-test-slave-44kr krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:05.513473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42345
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:05.591975 18432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:05.592185 18432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:05.592207 18432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:05.593375 18432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:05.593425 18432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:05.593438 18432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:05.593449 18432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:05.593461 18432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:05.594936 18432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:05.595162 18432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:05.595376 18432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:05.599998 18437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:05.600066 18438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:05.600068 18432 server_base.cc:1034] running on GCE node
W20260927 02:46:05.600167 18440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:05.600526 18432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:05.600818 18432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:05.601967 18432 hybrid_clock.cc:648] HybridClock initialized: now 1790477165601943 us; error 41 us; skew 500 ppm
Sep 27 02:46:05 dist-test-slave-44kr krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:05.604367 18432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:05.604841 18432 webserver.cc:460] Webserver started at http://127.0.98.62:33407/ using document root <none> and password file <none>
I20260927 02:46:05.605052 18432 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:05.605098 18432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:05.605193 18432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:05.606199 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "fa115bcd0b714710bb9f3af294d84ff6"
format_stamp: "Formatted at 2026-09-27 02:46:05 on dist-test-slave-44kr"
server_key: "954ccde2bec0a380aa895399b6e29a8f"
server_key_iv: "80cbe55ce20aa15d6908cae6a4eadf5e"
server_key_version: "encryptionkey@0"
I20260927 02:46:05.606477 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa115bcd0b714710bb9f3af294d84ff6"
format_stamp: "Formatted at 2026-09-27 02:46:05 on dist-test-slave-44kr"
server_key: "954ccde2bec0a380aa895399b6e29a8f"
server_key_iv: "80cbe55ce20aa15d6908cae6a4eadf5e"
server_key_version: "encryptionkey@0"
I20260927 02:46:05.607702 18432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:05.608372 18447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:05.608599 18432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:05.608680 18432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "fa115bcd0b714710bb9f3af294d84ff6"
format_stamp: "Formatted at 2026-09-27 02:46:05 on dist-test-slave-44kr"
server_key: "954ccde2bec0a380aa895399b6e29a8f"
server_key_iv: "80cbe55ce20aa15d6908cae6a4eadf5e"
server_key_version: "encryptionkey@0"
I20260927 02:46:05.608747 18432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:05.619455 18432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:05.621322 18432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:05.621455 18432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:05.625093 18432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42345
I20260927 02:46:05.625177 18499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42345 every 8 connection(s)
I20260927 02:46:05.625458 18432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:05.626286 18500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:05.627964 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 18432
I20260927 02:46:05.628082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:05.628286 392 external_mini_cluster.cc:1428] Setting key bf66e7c894ea89aa80a379b39cc8b0a5
I20260927 02:46:05.628780 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6: Bootstrap starting.
I20260927 02:46:05.629354 18500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:05.629601 18500 log.cc:824] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:05.630388 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6: No bootstrap required, opened a new log
I20260927 02:46:05.632342 18500 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa115bcd0b714710bb9f3af294d84ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42345 } }
I20260927 02:46:05.632539 18500 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:05.632570 18500 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa115bcd0b714710bb9f3af294d84ff6, State: Initialized, Role: FOLLOWER
I20260927 02:46:05.632695 18500 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa115bcd0b714710bb9f3af294d84ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42345 } }
I20260927 02:46:05.632757 18500 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:05.632788 18500 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:05.632833 18500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:05.633455 18500 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa115bcd0b714710bb9f3af294d84ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42345 } }
I20260927 02:46:05.633549 18500 leader_election.cc:302] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa115bcd0b714710bb9f3af294d84ff6; no voters:
I20260927 02:46:05.633703 18500 leader_election.cc:288] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:05.633728 18505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:05.633837 18505 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [term 1 LEADER]: Becoming Leader. State: Replica: fa115bcd0b714710bb9f3af294d84ff6, State: Running, Role: LEADER
I20260927 02:46:05.633950 18500 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:05.633976 18505 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa115bcd0b714710bb9f3af294d84ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42345 } }
I20260927 02:46:05.634326 18505 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa115bcd0b714710bb9f3af294d84ff6. Latest consensus state: current_term: 1 leader_uuid: "fa115bcd0b714710bb9f3af294d84ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa115bcd0b714710bb9f3af294d84ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42345 } } }
I20260927 02:46:05.634577 18505 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:05.634498 18507 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa115bcd0b714710bb9f3af294d84ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa115bcd0b714710bb9f3af294d84ff6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42345 } } }
I20260927 02:46:05.634965 18507 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:05.635123 18514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 02:46:05 dist-test-slave-44kr krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:05.635689 18514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:05.637197 18514 catalog_manager.cc:1347] Generated new cluster ID: cb7497e0c6c644aa8693611345901f7f
I20260927 02:46:05.637248 18514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:05.639242 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:05.628839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.0.98.62:42345)
0927 02:46:05.629026 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:05.629034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:05.629060 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:46:05.629621 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:05.629629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:05.629684 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:05.629843 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:05.630499 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:05.631427 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:05.632493 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:05.633064 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:05.633144 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:05.635850 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:05.635854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:05.635854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:05.635864 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:05.637805 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:05.638143 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:05.638146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:05.638146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:05.638234 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:05.638404 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:05.638405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:05.638405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:05.638636 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:05.638785 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:05.638890 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:46:05.639034 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260927 02:46:05.668725 18514 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:05.669286 18514 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:05.673803 18514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6: Generated new TSK 0
I20260927 02:46:05.673982 18514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:46:05.704326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42345
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:05.818784 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:05.819056 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:05.819092 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:05.820749 18526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:05.820804 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:05.820843 18526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:05.822762 18526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:05.823032 18526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:05.823292 18526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:05.828927 18531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:05.829030 18532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:05.828927 18534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:05.829278 18526 server_base.cc:1034] running on GCE node
I20260927 02:46:05.829470 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:05.829715 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:05.830885 18526 hybrid_clock.cc:648] HybridClock initialized: now 1790477165830870 us; error 29 us; skew 500 ppm
Sep 27 02:46:05 dist-test-slave-44kr krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:05.833485 18526 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:46:05.834007 18526 webserver.cc:460] Webserver started at http://127.0.98.1:41245/ using document root <none> and password file <none>
I20260927 02:46:05.834262 18526 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:05.834331 18526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:05.834504 18526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:05.835649 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "924a946a0a344f7abd312069b4dde8e8"
format_stamp: "Formatted at 2026-09-27 02:46:05 on dist-test-slave-44kr"
server_key: "7d447ff1149fd37c6dbb9160e0cf36eb"
server_key_iv: "cfa04976054a4146472783866c04149d"
server_key_version: "encryptionkey@0"
I20260927 02:46:05.835971 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "924a946a0a344f7abd312069b4dde8e8"
format_stamp: "Formatted at 2026-09-27 02:46:05 on dist-test-slave-44kr"
server_key: "7d447ff1149fd37c6dbb9160e0cf36eb"
server_key_iv: "cfa04976054a4146472783866c04149d"
server_key_version: "encryptionkey@0"
I20260927 02:46:05.837355 18526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:05.838045 18541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:05.838178 18526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:46:05.838260 18526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "924a946a0a344f7abd312069b4dde8e8"
format_stamp: "Formatted at 2026-09-27 02:46:05 on dist-test-slave-44kr"
server_key: "7d447ff1149fd37c6dbb9160e0cf36eb"
server_key_iv: "cfa04976054a4146472783866c04149d"
server_key_version: "encryptionkey@0"
I20260927 02:46:05.838327 18526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:05.862672 18526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:05.864812 18526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:05.864946 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:05.865170 18526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:05.865446 18526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:05.865476 18526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:05.865505 18526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:05.865523 18526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:05.870937 18526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43397
I20260927 02:46:05.871007 18654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43397 every 8 connection(s)
I20260927 02:46:05.871291 18526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:46:05 dist-test-slave-44kr krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:05.879958 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:05.872026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35845 (local address 127.0.98.62:42345)
0927 02:46:05.872113 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:05.872115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:05.872254 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:46:05.873168 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:05.873169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:05.873180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:05.873232 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:05.873826 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:05.874398 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:05.875373 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:05.875582 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:05.875635 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:05.877582 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:05.877583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:05.877584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:05.877591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:05.878772 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:05.879126 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:05.879127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:05.879127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:05.879177 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:05.879382 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:05.879383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:05.879383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:05.879485 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:05.879539 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:05.879792 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 02:46:05.879869 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:46:05.880121 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:05.871958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42345 (local address 127.0.98.1:35845)
0927 02:46:05.872121 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:05.872166 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:05.873082 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:05.873293 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:05.873296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:05.873538 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:05.873733 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:05.873737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:05.874469 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:05.874470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:05.875299 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:05.875301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:05.875645 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:05.875646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:05.875739 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:05.876180 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:05.876187 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:05.877498 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:05.878844 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:05.878851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:05.878853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:05.879055 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:05.879229 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:05.879229 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:05.879230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:05.879330 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:05.879537 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:05.879543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:05.879680 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 02:46:05.879765 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:46:05.879905 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260927 02:46:05.880149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 18526
I20260927 02:46:05.880246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:46:05.880494 392 external_mini_cluster.cc:1428] Setting key 576e55db3eb5f9564791bb4acae51cc1
I20260927 02:46:05.880816 18655 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42345
I20260927 02:46:05.880915 18655 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:05.881098 18655 heartbeater.cc:511] Master 127.0.98.62:42345 requested a full tablet report, sending...
I20260927 02:46:05.881610 18464 ts_manager.cc:194] Registered new tserver with Master: 924a946a0a344f7abd312069b4dde8e8 (127.0.98.1:43397)
I20260927 02:46:05.882236 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35845
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:46:05.903890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42345
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:05.983708 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:05.983937 18660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:05.983959 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:05.985133 18660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:05.985174 18660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:05.985199 18660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:05.986645 18660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:05.986836 18660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:05.987015 18660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:05.991940 18666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:05.991942 18665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:05.992089 18660 server_base.cc:1034] running on GCE node
W20260927 02:46:05.991945 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:05.992406 18660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:05.992652 18660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:05.993800 18660 hybrid_clock.cc:648] HybridClock initialized: now 1790477165993785 us; error 28 us; skew 500 ppm
Sep 27 02:46:05 dist-test-slave-44kr krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:05.996294 18660 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:46:05.996827 18660 webserver.cc:460] Webserver started at http://127.0.98.2:43959/ using document root <none> and password file <none>
I20260927 02:46:05.997088 18660 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:05.997150 18660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:05.997287 18660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:05.998379 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "c482976453b54dd29b508ec65479a684"
format_stamp: "Formatted at 2026-09-27 02:46:05 on dist-test-slave-44kr"
server_key: "21bd91506ef488c6177b7e10499a06d7"
server_key_iv: "a24053eb61adfcdbd3f2aa02d1bfebf7"
server_key_version: "encryptionkey@0"
I20260927 02:46:05.998694 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c482976453b54dd29b508ec65479a684"
format_stamp: "Formatted at 2026-09-27 02:46:05 on dist-test-slave-44kr"
server_key: "21bd91506ef488c6177b7e10499a06d7"
server_key_iv: "a24053eb61adfcdbd3f2aa02d1bfebf7"
server_key_version: "encryptionkey@0"
I20260927 02:46:05.999946 18660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:46:06.000777 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.000978 18660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.001060 18660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "c482976453b54dd29b508ec65479a684"
format_stamp: "Formatted at 2026-09-27 02:46:05 on dist-test-slave-44kr"
server_key: "21bd91506ef488c6177b7e10499a06d7"
server_key_iv: "a24053eb61adfcdbd3f2aa02d1bfebf7"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.001128 18660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:06.025117 18660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:06.027288 18660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:06.027412 18660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:06.027648 18660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:06.027971 18660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:06.028004 18660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.028033 18660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:06.028050 18660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.033507 18660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45115
I20260927 02:46:06.033569 18788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45115 every 8 connection(s)
I20260927 02:46:06.033874 18660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:06.038792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 18660
I20260927 02:46:06.038893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:46:06.039144 392 external_mini_cluster.cc:1428] Setting key 0b97bb7a44dea2ec3d51543a63b02cfd
Sep 27 02:46:06 dist-test-slave-44kr krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:06.043684 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.034658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49911 (local address 127.0.98.62:42345)
0927 02:46:06.034733 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.034735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.035022 (+ 287us) server_negotiation.cc:407] Connection header received
0927 02:46:06.035898 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.035899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.035909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.035974 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:06.036575 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.037108 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.038121 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.038312 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.038374 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.040630 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:06.040631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:06.040631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:06.040639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:06.042192 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:06.042646 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:06.042647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:06.042647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:06.042707 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:06.043013 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:06.043014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:06.043014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:06.043131 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:06.043192 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.043534 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.043618 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:46:06.043898 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.034610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42345 (local address 127.0.98.2:49911)
0927 02:46:06.034778 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.034821 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.035794 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.036036 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.036039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.036281 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:06.036495 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.036501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.037184 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.037184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.038010 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.038012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.038407 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.038408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.038528 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.039055 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:06.039064 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:06.040546 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:06.042263 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:06.042270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:06.042273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:06.042547 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:06.042772 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:06.042773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:06.042773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:06.042901 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:06.043195 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:06.043201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:06.043366 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.043484 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.043665 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260927 02:46:06.044377 18789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42345
I20260927 02:46:06.044502 18789 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:06.044675 18789 heartbeater.cc:511] Master 127.0.98.62:42345 requested a full tablet report, sending...
I20260927 02:46:06.045087 18464 ts_manager.cc:194] Registered new tserver with Master: c482976453b54dd29b508ec65479a684 (127.0.98.2:45115)
I20260927 02:46:06.045459 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49911
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:46:06.065354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42345
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:06.145030 18794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:06.145278 18794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:06.145303 18794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:06.146588 18794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:06.146641 18794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:06.146668 18794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:06.148258 18794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:06.148523 18794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:06.148751 18794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:06.153839 18802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:06.153872 18794 server_base.cc:1034] running on GCE node
W20260927 02:46:06.153849 18800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:06.154204 18799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:06.154513 18794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:06.154769 18794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:06.155915 18794 hybrid_clock.cc:648] HybridClock initialized: now 1790477166155900 us; error 31 us; skew 500 ppm
Sep 27 02:46:06 dist-test-slave-44kr krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:06.158538 18794 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:46:06.159017 18794 webserver.cc:460] Webserver started at http://127.0.98.3:38267/ using document root <none> and password file <none>
I20260927 02:46:06.159228 18794 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:06.159276 18794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:06.159405 18794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:06.160544 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "8dfc74d162c64a6d8705d5a0b33e44db"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "9f4811177cd6d1f8946b54159514f8f5"
server_key_iv: "dfe2a44a63206994e8f4d69637a6005b"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.160884 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8dfc74d162c64a6d8705d5a0b33e44db"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "9f4811177cd6d1f8946b54159514f8f5"
server_key_iv: "dfe2a44a63206994e8f4d69637a6005b"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.162304 18794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:46:06.163230 18809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.163471 18794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:06.163559 18794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "8dfc74d162c64a6d8705d5a0b33e44db"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "9f4811177cd6d1f8946b54159514f8f5"
server_key_iv: "dfe2a44a63206994e8f4d69637a6005b"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.163626 18794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:06.198040 18794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:06.200177 18794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:06.200318 18794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:06.200572 18794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:06.200865 18794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:06.200901 18794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.200929 18794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:06.200947 18794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.206333 18794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34757
I20260927 02:46:06.206403 18922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34757 every 8 connection(s)
I20260927 02:46:06.206694 18794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:06.210391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 18794
I20260927 02:46:06.210494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:46:06.210671 392 external_mini_cluster.cc:1428] Setting key b5623b3d56fcfbd2be417e3fbf3ed2df
Sep 27 02:46:06 dist-test-slave-44kr krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:06.216481 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.207408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59645 (local address 127.0.98.62:42345)
0927 02:46:06.207472 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.207474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.207697 (+ 223us) server_negotiation.cc:407] Connection header received
0927 02:46:06.208638 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.208639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.208650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.208708 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:06.209328 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.209838 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.211049 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.211302 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.211361 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.213667 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:06.213669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:06.213670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:06.213678 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:06.215079 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:06.215512 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:06.215513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:06.215513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:06.215576 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:06.215825 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:06.215826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:06.215826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:06.215946 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:06.216002 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.216267 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.216351 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:46:06.216684 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.207369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42345 (local address 127.0.98.3:59645)
0927 02:46:06.207527 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.207565 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.208523 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.208770 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.208774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.209031 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:06.209240 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.209245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.209908 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.209908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.210989 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.210992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.211369 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.211370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.211535 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.212058 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:06.212066 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:06.213558 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:06.215203 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:06.215212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:06.215215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:06.215441 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:06.215660 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:06.215661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:06.215661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:06.215766 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:06.216000 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:06.216007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:06.216148 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.216238 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.216453 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260927 02:46:06.217170 18923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42345
I20260927 02:46:06.217264 18923 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:06.217435 18923 heartbeater.cc:511] Master 127.0.98.62:42345 requested a full tablet report, sending...
I20260927 02:46:06.217801 18464 ts_manager.cc:194] Registered new tserver with Master: 8dfc74d162c64a6d8705d5a0b33e44db (127.0.98.3:34757)
I20260927 02:46:06.218201 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59645
I20260927 02:46:06.222460 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:06.229998 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.223787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.0.98.62:42345)
0927 02:46:06.223861 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.223863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.223906 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:46:06.223970 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.223971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.223980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.224025 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:06.224609 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.225134 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.225897 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.226082 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.226135 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.227634 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:06.227635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:06.227636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:06.227644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:06.228907 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:06.229276 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:06.229277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:06.229277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:06.229327 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:06.229508 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:06.229509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:06.229509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:06.229643 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:06.229740 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.229850 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.229908 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:46:06.231771 18464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:06.234069 18464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:06.241169 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.236948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43397 (local address 127.0.0.1:52498)
0927 02:46:06.237478 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.237487 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.237538 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.237771 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.237772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.237774 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.237821 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.237823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.238732 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.238733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.239988 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.239990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.240855 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.240856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.240930 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.240946 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.241010 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.241078 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260927 02:46:06.241200 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.237187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.0.98.3:34757)
0927 02:46:06.237380 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.237383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.237390 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:46:06.237424 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.237425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.237437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.237511 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.237893 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.239006 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.240268 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.240898 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.240967 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.241024 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.241091 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.241128 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260927 02:46:06.241225 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.237137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.0.98.2:45115)
0927 02:46:06.237313 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.237318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.237682 (+ 364us) server_negotiation.cc:407] Connection header received
0927 02:46:06.237725 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.237727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.237741 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.237809 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.237932 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.239005 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.240002 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.240770 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.240840 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.240900 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.241107 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.241146 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260927 02:46:06.241243 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.236948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34757 (local address 127.0.0.1:56016)
0927 02:46:06.237130 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.237171 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.237241 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.237527 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.237531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.237534 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.237621 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.237623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.239129 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.239130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.240199 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.240201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.240964 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.240965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.241015 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.241017 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.241076 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.241117 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":45,"thread_start_us":69,"threads_started":1}
I20260927 02:46:06.241598 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.237326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.0.98.1:43397)
0927 02:46:06.237473 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.237478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.237528 (+ 50us) server_negotiation.cc:407] Connection header received
0927 02:46:06.237608 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.237610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.237626 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.237697 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.237895 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.238641 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.240046 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.240776 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.240841 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.240900 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.241441 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.241497 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260927 02:46:06.241627 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.236950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45115 (local address 127.0.0.1:57652)
0927 02:46:06.237617 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.237624 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.237667 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.237811 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.237811 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.237813 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.237855 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.237856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.239122 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.239122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.239921 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.239924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.240935 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.240936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.241001 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.241005 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.241522 (+ 517us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.241566 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":42,"thread_start_us":113,"threads_started":1}
I20260927 02:46:06.242015 18857 tablet_service.cc:1467] Processing CreateTablet for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9 (DEFAULT_TABLE table=test-table [id=4661cd5a7ba94604a199c30fd631c758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:06.242168 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9 (DEFAULT_TABLE table=test-table [id=4661cd5a7ba94604a199c30fd631c758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:06.242322 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:06.242476 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:06.244827 18940 tablet_bootstrap.cc:492] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: Bootstrap starting.
I20260927 02:46:06.244844 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9 (DEFAULT_TABLE table=test-table [id=4661cd5a7ba94604a199c30fd631c758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:06.245112 18941 tablet_bootstrap.cc:492] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8: Bootstrap starting.
I20260927 02:46:06.245190 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:06.245549 18940 tablet_bootstrap.cc:654] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:06.245703 18941 tablet_bootstrap.cc:654] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:06.246042 18940 log.cc:824] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:06.246213 18941 log.cc:824] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:06.246727 18940 tablet_bootstrap.cc:492] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: No bootstrap required, opened a new log
I20260927 02:46:06.246788 18940 ts_tablet_manager.cc:1397] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:06.246927 18941 tablet_bootstrap.cc:492] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8: No bootstrap required, opened a new log
I20260927 02:46:06.246997 18941 ts_tablet_manager.cc:1397] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:06.248243 18944 tablet_bootstrap.cc:492] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: Bootstrap starting.
I20260927 02:46:06.248143 18940 raft_consensus.cc:348] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260927 02:46:06.248309 18940 raft_consensus.cc:374] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:06.248347 18940 raft_consensus.cc:729] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfc74d162c64a6d8705d5a0b33e44db, State: Initialized, Role: FOLLOWER
I20260927 02:46:06.248479 18940 consensus_queue.cc:260] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260927 02:46:06.248776 18940 ts_tablet_manager.cc:1428] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:06.248812 18923 heartbeater.cc:503] Master 127.0.98.62:42345 was elected leader, sending a full tablet report...
I20260927 02:46:06.249189 18944 tablet_bootstrap.cc:654] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:06.249138 18941 raft_consensus.cc:348] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260927 02:46:06.249292 18941 raft_consensus.cc:374] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:06.249325 18941 raft_consensus.cc:729] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924a946a0a344f7abd312069b4dde8e8, State: Initialized, Role: FOLLOWER
I20260927 02:46:06.249441 18941 consensus_queue.cc:260] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260927 02:46:06.249706 18941 ts_tablet_manager.cc:1428] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:06.249696 18944 log.cc:824] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:06.249752 18655 heartbeater.cc:503] Master 127.0.98.62:42345 was elected leader, sending a full tablet report...
I20260927 02:46:06.250437 18944 tablet_bootstrap.cc:492] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: No bootstrap required, opened a new log
I20260927 02:46:06.250487 18944 ts_tablet_manager.cc:1397] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:06.251890 18944 raft_consensus.cc:348] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260927 02:46:06.252022 18944 raft_consensus.cc:374] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:06.252043 18944 raft_consensus.cc:729] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c482976453b54dd29b508ec65479a684, State: Initialized, Role: FOLLOWER
I20260927 02:46:06.252139 18944 consensus_queue.cc:260] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260927 02:46:06.252374 18944 ts_tablet_manager.cc:1428] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:06.252435 18789 heartbeater.cc:503] Master 127.0.98.62:42345 was elected leader, sending a full tablet report...
W20260927 02:46:06.284545 18790 tablet.cc:2369] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:06.290877 18946 raft_consensus.cc:482] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:06.291000 18946 raft_consensus.cc:504] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260927 02:46:06.291329 18946 leader_election.cc:288] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c482976453b54dd29b508ec65479a684 (127.0.98.2:45115), 8dfc74d162c64a6d8705d5a0b33e44db (127.0.98.3:34757)
I20260927 02:46:06.294102 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.291619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38669 (local address 127.0.98.2:45115)
0927 02:46:06.291669 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.291671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.291679 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:06.291706 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.291707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.291720 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.291784 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.291935 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.292462 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.293444 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.293797 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.293852 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.293862 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.293997 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.294033 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:46:06.294093 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.291493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45115 (local address 127.0.98.1:38669)
0927 02:46:06.291543 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.291556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.291635 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.291794 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.291795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.291796 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.291872 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.291874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.292527 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.292528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.293354 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.293357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.293868 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.293869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.293937 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.293940 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.293978 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.294010 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 02:46:06.294131 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.291619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56313 (local address 127.0.98.3:34757)
0927 02:46:06.291660 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.291662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.291802 (+ 140us) server_negotiation.cc:407] Connection header received
0927 02:46:06.291837 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.291838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.291848 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.291903 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.292040 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.292562 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.293444 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.293804 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.293864 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.293873 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.294026 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.294066 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:46:06.294286 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.291523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34757 (local address 127.0.98.1:56313)
0927 02:46:06.291729 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.291738 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.291794 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.291913 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.291913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.291915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.291983 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.291985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.292635 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.292636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.293363 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.293365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.293880 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.293881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.293960 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.293963 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.294014 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.294050 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":116,"threads_started":1}
I20260927 02:46:06.294656 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f0b0d3d5ae43c8ae89575e14ebadd9" candidate_uuid: "924a946a0a344f7abd312069b4dde8e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" is_pre_election: true
I20260927 02:46:06.294657 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f0b0d3d5ae43c8ae89575e14ebadd9" candidate_uuid: "924a946a0a344f7abd312069b4dde8e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c482976453b54dd29b508ec65479a684" is_pre_election: true
I20260927 02:46:06.294818 18743 raft_consensus.cc:2454] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924a946a0a344f7abd312069b4dde8e8 in term 0.
I20260927 02:46:06.294818 18877 raft_consensus.cc:2454] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924a946a0a344f7abd312069b4dde8e8 in term 0.
I20260927 02:46:06.295029 18544 leader_election.cc:302] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfc74d162c64a6d8705d5a0b33e44db, 924a946a0a344f7abd312069b4dde8e8; no voters:
I20260927 02:46:06.295208 18946 raft_consensus.cc:2781] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:06.295264 18946 raft_consensus.cc:482] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:06.295285 18946 raft_consensus.cc:3037] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:06.295905 18946 raft_consensus.cc:504] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260927 02:46:06.296015 18946 leader_election.cc:288] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [CANDIDATE]: Term 1 election: Requested vote from peers c482976453b54dd29b508ec65479a684 (127.0.98.2:45115), 8dfc74d162c64a6d8705d5a0b33e44db (127.0.98.3:34757)
I20260927 02:46:06.296154 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f0b0d3d5ae43c8ae89575e14ebadd9" candidate_uuid: "924a946a0a344f7abd312069b4dde8e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfc74d162c64a6d8705d5a0b33e44db"
I20260927 02:46:06.296172 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f0b0d3d5ae43c8ae89575e14ebadd9" candidate_uuid: "924a946a0a344f7abd312069b4dde8e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c482976453b54dd29b508ec65479a684"
I20260927 02:46:06.296231 18877 raft_consensus.cc:3037] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:06.296231 18743 raft_consensus.cc:3037] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:06.297097 18877 raft_consensus.cc:2454] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924a946a0a344f7abd312069b4dde8e8 in term 1.
I20260927 02:46:06.297092 18743 raft_consensus.cc:2454] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924a946a0a344f7abd312069b4dde8e8 in term 1.
I20260927 02:46:06.297256 18544 leader_election.cc:302] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfc74d162c64a6d8705d5a0b33e44db, 924a946a0a344f7abd312069b4dde8e8; no voters:
I20260927 02:46:06.297370 18946 raft_consensus.cc:2781] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:06.297518 18946 raft_consensus.cc:686] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [term 1 LEADER]: Becoming Leader. State: Replica: 924a946a0a344f7abd312069b4dde8e8, State: Running, Role: LEADER
I20260927 02:46:06.297624 18946 consensus_queue.cc:237] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260927 02:46:06.298470 18464 catalog_manager.cc:5495] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 924a946a0a344f7abd312069b4dde8e8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "924a946a0a344f7abd312069b4dde8e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924a946a0a344f7abd312069b4dde8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } health_report { overall_health: UNKNOWN } } }
I20260927 02:46:06.305176 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.302732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.0.98.1:43397)
0927 02:46:06.302786 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.302788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.302795 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:46:06.302872 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.302873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.302888 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.302944 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:06.303134 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.303627 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.304432 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.304643 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.304699 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.304862 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:06.304952 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:06.305037 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.305080 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.305090 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:46:06.306883 18743 raft_consensus.cc:1261] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [term 1 FOLLOWER]: Refusing update from remote peer 924a946a0a344f7abd312069b4dde8e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:06.306883 18877 raft_consensus.cc:1261] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [term 1 FOLLOWER]: Refusing update from remote peer 924a946a0a344f7abd312069b4dde8e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:06.307235 18950 consensus_queue.cc:1035] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c482976453b54dd29b508ec65479a684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:06.307375 18946 consensus_queue.cc:1035] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 924a946a0a344f7abd312069b4dde8e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfc74d162c64a6d8705d5a0b33e44db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:06.310024 18955 mvcc.cc:205] Tried to move back new op lower bound from 7333794473190137856 to 7333794473155719168. Current Snapshot: MvccSnapshot[applied={T|T < 7333794473190137856}]
I20260927 02:46:06.310356 18954 mvcc.cc:205] Tried to move back new op lower bound from 7333794473190137856 to 7333794473155719168. Current Snapshot: MvccSnapshot[applied={T|T < 7333794473190137856 or (T in {7333794473190137856})}]
I20260927 02:46:06.311563 18464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:06.311658 18464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:06.312361 18464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa115bcd0b714710bb9f3af294d84ff6: Sending DeleteTablet for 3 replicas of tablet 98f0b0d3d5ae43c8ae89575e14ebadd9
I20260927 02:46:06.312664 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52498
I20260927 02:46:06.312680 18723 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57652
I20260927 02:46:06.312762 18857 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56016
I20260927 02:46:06.312978 18963 tablet_replica.cc:330] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: stopping tablet replica
I20260927 02:46:06.313071 18963 raft_consensus.cc:2229] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:06.313056 18964 tablet_replica.cc:330] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: stopping tablet replica
I20260927 02:46:06.313148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 18526
I20260927 02:46:06.313159 18964 raft_consensus.cc:2229] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:06.313182 18963 raft_consensus.cc:2258] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:06.313247 18964 raft_consensus.cc:2258] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:06.313675 18963 ts_tablet_manager.cc:1905] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:06.314854 18964 ts_tablet_manager.cc:1905] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:06.315389 18963 ts_tablet_manager.cc:1918] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:06.315454 18963 log.cc:1196] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/98f0b0d3d5ae43c8ae89575e14ebadd9
I20260927 02:46:06.315686 18963 ts_tablet_manager.cc:1939] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P c482976453b54dd29b508ec65479a684: Deleting consensus metadata
I20260927 02:46:06.316008 18964 ts_tablet_manager.cc:1918] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:06.316052 18964 log.cc:1196] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/98f0b0d3d5ae43c8ae89575e14ebadd9
I20260927 02:46:06.316241 18964 ts_tablet_manager.cc:1939] T 98f0b0d3d5ae43c8ae89575e14ebadd9 P 8dfc74d162c64a6d8705d5a0b33e44db: Deleting consensus metadata
I20260927 02:46:06.316322 18451 catalog_manager.cc:4841] TS c482976453b54dd29b508ec65479a684 (127.0.98.2:45115): tablet 98f0b0d3d5ae43c8ae89575e14ebadd9 (table test-table [id=4661cd5a7ba94604a199c30fd631c758]) successfully deleted
I20260927 02:46:06.316725 18450 catalog_manager.cc:4841] TS 8dfc74d162c64a6d8705d5a0b33e44db (127.0.98.3:34757): tablet 98f0b0d3d5ae43c8ae89575e14ebadd9 (table test-table [id=4661cd5a7ba94604a199c30fd631c758]) successfully deleted
W20260927 02:46:06.318418 18448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43397 (error 108)
I20260927 02:46:06.318653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 18660
I20260927 02:46:06.319571 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.319463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43397 (local address 127.0.0.1:52514)
0927 02:46:06.319510 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.319543 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 02:46:06.319680 18448 catalog_manager.cc:4584] TS 924a946a0a344f7abd312069b4dde8e8 (127.0.98.1:43397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98f0b0d3d5ae43c8ae89575e14ebadd9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43397: connect: Connection refused (error 111)
I20260927 02:46:06.322908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 18794
I20260927 02:46:06.326633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 18432
2026-09-27T02:46:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1873 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T02:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:06Z Disabled control of system clock
I20260927 02:46:06.351821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39637
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:46:06.430787 18969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:06.431015 18969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:06.431033 18969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:06.432185 18969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:06.432233 18969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:06.432245 18969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:06.432255 18969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:06.432266 18969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:06.433645 18969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:06.433884 18969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:06.434180 18969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:06.438937 18974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:06.438957 18975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:06.439109 18969 server_base.cc:1034] running on GCE node
W20260927 02:46:06.438958 18977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:06.439482 18969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:06.439783 18969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:06.440927 18969 hybrid_clock.cc:648] HybridClock initialized: now 1790477166440908 us; error 39 us; skew 500 ppm
I20260927 02:46:06.442127 18969 webserver.cc:460] Webserver started at http://127.0.98.62:39719/ using document root <none> and password file <none>
I20260927 02:46:06.442340 18969 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:06.442386 18969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:06.442507 18969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:06.443565 18969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "02cdd407068640a68fba3264011a72a8"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "f04ccc1aaf5f1dea3ffa93e04cc18c67"
server_key_iv: "90957914e287ea4d4691dc9d783f2c01"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.443888 18969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "02cdd407068640a68fba3264011a72a8"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "f04ccc1aaf5f1dea3ffa93e04cc18c67"
server_key_iv: "90957914e287ea4d4691dc9d783f2c01"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.445207 18969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:06.445986 18983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.446198 18969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:06.446286 18969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "02cdd407068640a68fba3264011a72a8"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "f04ccc1aaf5f1dea3ffa93e04cc18c67"
server_key_iv: "90957914e287ea4d4691dc9d783f2c01"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.446357 18969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:06.462318 18969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:06.462613 18969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:06.462746 18969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:06.466684 18969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39637
I20260927 02:46:06.466720 19035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39637 every 8 connection(s)
I20260927 02:46:06.467048 18969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:06.467935 19036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:06.470175 19036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8: Bootstrap starting.
I20260927 02:46:06.470697 19036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:06.470923 19036 log.cc:824] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:06.471611 19036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8: No bootstrap required, opened a new log
I20260927 02:46:06.473235 19036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cdd407068640a68fba3264011a72a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39637 } }
I20260927 02:46:06.473368 19036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:06.473389 19036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02cdd407068640a68fba3264011a72a8, State: Initialized, Role: FOLLOWER
I20260927 02:46:06.473505 19036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cdd407068640a68fba3264011a72a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39637 } }
I20260927 02:46:06.473575 19036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:06.473604 19036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:06.473635 19036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:06.474102 19036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cdd407068640a68fba3264011a72a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39637 } }
I20260927 02:46:06.474182 19036 leader_election.cc:302] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02cdd407068640a68fba3264011a72a8; no voters:
I20260927 02:46:06.474301 19036 leader_election.cc:288] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:06.474376 19039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:06.474531 19036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:06.474555 19039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [term 1 LEADER]: Becoming Leader. State: Replica: 02cdd407068640a68fba3264011a72a8, State: Running, Role: LEADER
I20260927 02:46:06.474666 19039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cdd407068640a68fba3264011a72a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39637 } }
I20260927 02:46:06.475082 19040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02cdd407068640a68fba3264011a72a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cdd407068640a68fba3264011a72a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39637 } } }
I20260927 02:46:06.475157 19040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:06.475152 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02cdd407068640a68fba3264011a72a8. Latest consensus state: current_term: 1 leader_uuid: "02cdd407068640a68fba3264011a72a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cdd407068640a68fba3264011a72a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39637 } } }
I20260927 02:46:06.475242 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:06.475569 19048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:06.476034 19048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:06.476080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 18969
I20260927 02:46:06.476190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:06.476377 392 external_mini_cluster.cc:1428] Setting key da66e630857537c015d0b9ca66eba64d
I20260927 02:46:06.477600 19048 catalog_manager.cc:1347] Generated new cluster ID: 7be1864ae6804b19b16a126e5ff6c0a4
I20260927 02:46:06.477654 19048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:06.481221 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.476927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.0.98.62:39637)
0927 02:46:06.477149 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.477159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.477195 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:46:06.477330 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.477336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.477388 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.477500 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:06.477754 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.478646 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.479411 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.479825 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.479941 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.480585 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:06.480589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:06.480589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:06.480753 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:06.480786 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:06.480797 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:06.480851 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.480929 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.481018 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260927 02:46:06.482733 19048 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:06.483224 19048 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:06.491227 19048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8: Generated new TSK 0
I20260927 02:46:06.491402 19048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:06.497376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39637
--builtin_ntp_servers=127.0.98.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:46:06.578962 19060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:06.579205 19060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:06.579236 19060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:06.580410 19060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:06.580459 19060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:06.580495 19060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:06.581837 19060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:06.582072 19060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:06.582340 19060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:46:06.588364 19060 server_base.cc:1034] running on GCE node
W20260927 02:46:06.588347 19066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:06.588347 19068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:06.588321 19065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:06.588868 19060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:06.589175 19060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:06.590343 19060 hybrid_clock.cc:648] HybridClock initialized: now 1790477166590331 us; error 29 us; skew 500 ppm
I20260927 02:46:06.591825 19060 webserver.cc:460] Webserver started at http://127.0.98.1:37113/ using document root <none> and password file <none>
I20260927 02:46:06.592087 19060 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:06.592144 19060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:06.592262 19060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:06.593699 19060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "eea84889e02b413199a570ee8decbac0"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "894b5696285872a7d21dadf19f93d9b9"
server_key_iv: "abdea4c2e9e6c9ac5e14ab0503a81afa"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.594110 19060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eea84889e02b413199a570ee8decbac0"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "894b5696285872a7d21dadf19f93d9b9"
server_key_iv: "abdea4c2e9e6c9ac5e14ab0503a81afa"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.595669 19060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:46:06.596552 19074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.596700 19060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:46:06.596784 19060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "eea84889e02b413199a570ee8decbac0"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "894b5696285872a7d21dadf19f93d9b9"
server_key_iv: "abdea4c2e9e6c9ac5e14ab0503a81afa"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.596851 19060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:06.614346 19060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:06.614660 19060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:06.614778 19060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:06.614995 19060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:06.615304 19060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:06.615336 19060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.615357 19060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:06.615368 19060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.620666 19060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45021
I20260927 02:46:06.620715 19187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45021 every 8 connection(s)
I20260927 02:46:06.621075 19060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:06.623219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 19060
I20260927 02:46:06.623325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:46:06.623503 392 external_mini_cluster.cc:1428] Setting key a3617cbc0272588df83787dbb5b9f393
I20260927 02:46:06.624428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39637
--builtin_ntp_servers=127.0.98.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:46:06.626857 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.621909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60821 (local address 127.0.98.62:39637)
0927 02:46:06.621977 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.621979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.622311 (+ 332us) server_negotiation.cc:407] Connection header received
0927 02:46:06.622838 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.622839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.622853 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.622909 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:06.623713 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.624375 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.625478 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.625669 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.625724 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.626597 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:06.626599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:06.626599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:06.626605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:06.626619 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:06.626623 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:06.626666 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.626745 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.626775 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:46:06.627128 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.621890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39637 (local address 127.0.98.1:60821)
0927 02:46:06.622183 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.622221 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.622722 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.622992 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.622996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.623009 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:06.623621 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.623627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.624510 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.624511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.625385 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.625387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.625741 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.625742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.625884 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.626448 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:46:06.626449 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:06.626479 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:46:06.626503 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:06.626672 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:06.626678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:06.626681 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.626742 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.626847 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260927 02:46:06.627607 19188 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39637
I20260927 02:46:06.627717 19188 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:06.627921 19188 heartbeater.cc:511] Master 127.0.98.62:39637 requested a full tablet report, sending...
I20260927 02:46:06.628556 19000 ts_manager.cc:194] Registered new tserver with Master: eea84889e02b413199a570ee8decbac0 (127.0.98.1:45021)
I20260927 02:46:06.629492 19000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60821
W20260927 02:46:06.705107 19191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:06.705325 19191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:06.705348 19191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:06.706475 19191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:06.706509 19191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:06.706533 19191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:06.707830 19191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:06.708014 19191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:06.708261 19191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:06.713131 19196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:06.713259 19199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:06.713387 19197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:06.713310 19191 server_base.cc:1034] running on GCE node
I20260927 02:46:06.713629 19191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:06.713862 19191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:06.715010 19191 hybrid_clock.cc:648] HybridClock initialized: now 1790477166714994 us; error 28 us; skew 500 ppm
I20260927 02:46:06.720487 19191 webserver.cc:460] Webserver started at http://127.0.98.2:40427/ using document root <none> and password file <none>
I20260927 02:46:06.720746 19191 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:06.720799 19191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:06.720923 19191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:06.721961 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "62cf7de7d9564d84a616aeca58046f08"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "c0a9ddbbd8f95a20e6cda0e1df7de586"
server_key_iv: "d03054f4329d3f860b3be2935f901e34"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.722280 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "62cf7de7d9564d84a616aeca58046f08"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "c0a9ddbbd8f95a20e6cda0e1df7de586"
server_key_iv: "d03054f4329d3f860b3be2935f901e34"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.723601 19191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:46:06.724303 19205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.724472 19191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:46:06.724556 19191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "62cf7de7d9564d84a616aeca58046f08"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "c0a9ddbbd8f95a20e6cda0e1df7de586"
server_key_iv: "d03054f4329d3f860b3be2935f901e34"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.724637 19191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:06.745277 19191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:06.745577 19191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:06.745712 19191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:06.745927 19191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:06.746254 19191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:06.746289 19191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.746315 19191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:06.746340 19191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.751849 19191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34565
I20260927 02:46:06.751919 19318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34565 every 8 connection(s)
I20260927 02:46:06.752218 19191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:06.757318 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.752932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32817 (local address 127.0.98.62:39637)
0927 02:46:06.753000 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.753002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.753217 (+ 215us) server_negotiation.cc:407] Connection header received
0927 02:46:06.753726 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.753727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.753737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.753792 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:06.754401 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.755001 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.755930 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.756160 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.756213 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.757061 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:06.757062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:06.757062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:06.757092 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:06.757105 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:06.757109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:06.757161 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.757233 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.757256 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:46:06.757495 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.752920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39637 (local address 127.0.98.2:32817)
0927 02:46:06.753107 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.753144 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.753632 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.753846 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.753849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.753857 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:06.754326 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.754330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.755077 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.755078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.755857 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.755859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.756242 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.756243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.756363 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.756915 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:46:06.756916 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:06.756946 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:46:06.756970 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:06.757167 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:06.757170 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:06.757172 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.757218 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.757296 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":126,"threads_started":1}
I20260927 02:46:06.757989 19319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39637
I20260927 02:46:06.758085 19319 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:06.758268 19319 heartbeater.cc:511] Master 127.0.98.62:39637 requested a full tablet report, sending...
I20260927 02:46:06.758670 19000 ts_manager.cc:194] Registered new tserver with Master: 62cf7de7d9564d84a616aeca58046f08 (127.0.98.2:34565)
I20260927 02:46:06.759083 19000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:32817
I20260927 02:46:06.759663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 19191
I20260927 02:46:06.759771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:46:06.759971 392 external_mini_cluster.cc:1428] Setting key ea83f791f2d3700acce78acbf557cfac
I20260927 02:46:06.760985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39637
--builtin_ntp_servers=127.0.98.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:46:06.842897 19322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:06.843112 19322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:06.843132 19322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:06.844281 19322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:06.844321 19322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:06.844345 19322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:06.845745 19322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:06.845978 19322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:06.846258 19322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:06.851100 19330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:06.851277 19327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:06.851397 19328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:06.851308 19322 server_base.cc:1034] running on GCE node
I20260927 02:46:06.851647 19322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:06.851897 19322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:06.853045 19322 hybrid_clock.cc:648] HybridClock initialized: now 1790477166853033 us; error 29 us; skew 500 ppm
I20260927 02:46:06.854323 19322 webserver.cc:460] Webserver started at http://127.0.98.3:35529/ using document root <none> and password file <none>
I20260927 02:46:06.854547 19322 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:06.854599 19322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:06.854733 19322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:06.855754 19322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "81221453becc4bf9ba92bdea0398c74b"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "a96ea38b9c7b5e8a6dc7e94c1458085c"
server_key_iv: "9ec9b6ed1c4f25626fd11f3c31c2055e"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.856081 19322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "81221453becc4bf9ba92bdea0398c74b"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "a96ea38b9c7b5e8a6dc7e94c1458085c"
server_key_iv: "9ec9b6ed1c4f25626fd11f3c31c2055e"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.857335 19322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:06.858032 19336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.858165 19322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:46:06.858249 19322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "81221453becc4bf9ba92bdea0398c74b"
format_stamp: "Formatted at 2026-09-27 02:46:06 on dist-test-slave-44kr"
server_key: "a96ea38b9c7b5e8a6dc7e94c1458085c"
server_key_iv: "9ec9b6ed1c4f25626fd11f3c31c2055e"
server_key_version: "encryptionkey@0"
I20260927 02:46:06.858304 19322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:06.876307 19322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:06.876664 19322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:06.876792 19322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:06.877014 19322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:06.877319 19322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:06.877352 19322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.877383 19322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:06.877403 19322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:06.882800 19322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45085
I20260927 02:46:06.882925 19449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45085 every 8 connection(s)
I20260927 02:46:06.883181 19322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:06.885643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 19322
I20260927 02:46:06.885741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:46:06.885916 392 external_mini_cluster.cc:1428] Setting key 834489a1b65174a047edc3663e722276
I20260927 02:46:06.888844 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.883953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56623 (local address 127.0.98.62:39637)
0927 02:46:06.884023 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.884024 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.884263 (+ 239us) server_negotiation.cc:407] Connection header received
0927 02:46:06.884811 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.884813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.884825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.884894 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:06.885579 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.886223 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.887394 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.887617 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.887680 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.888580 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:06.888581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:06.888581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:06.888585 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:06.888601 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:06.888604 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:06.888657 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.888756 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.888784 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:46:06.888983 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.883918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39637 (local address 127.0.98.3:56623)
0927 02:46:06.884146 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.884181 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.884709 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.884978 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.884981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.884989 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:06.885487 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.885492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.886309 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.886310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.887297 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.887300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.887712 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.887713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.887838 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.888374 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:46:06.888374 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:06.888438 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:46:06.888468 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:06.888661 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:06.888664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:06.888666 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.888721 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.888803 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":123,"threads_started":1}
I20260927 02:46:06.889446 19450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39637
I20260927 02:46:06.889528 19450 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:06.889668 19450 heartbeater.cc:511] Master 127.0.98.62:39637 requested a full tablet report, sending...
I20260927 02:46:06.889973 19000 ts_manager.cc:194] Registered new tserver with Master: 81221453becc4bf9ba92bdea0398c74b (127.0.98.3:45085)
I20260927 02:46:06.890322 19000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56623
I20260927 02:46:06.897521 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:06.902015 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.898974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.0.98.62:39637)
0927 02:46:06.899059 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.899061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.899125 (+ 64us) server_negotiation.cc:407] Connection header received
0927 02:46:06.899145 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.899146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.899156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.899201 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:06.899373 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.899942 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.900796 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.901024 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.901137 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.901774 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:06.901775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:06.901775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:06.901779 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:06.901791 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:06.901795 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:06.901862 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.901925 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.901934 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:46:06.903620 19000 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:06.905237 19000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:06.911872 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.907911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.0.98.1:45021)
0927 02:46:06.908112 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.908116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.908129 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:46:06.908174 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.908176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.908192 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.908275 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.908508 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.909641 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.910999 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.911487 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.911550 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.911655 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.911744 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.911781 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260927 02:46:06.911863 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.907672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45021 (local address 127.0.0.1:45856)
0927 02:46:06.908064 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.908072 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.908117 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.908348 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.908349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.908351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.908431 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.908433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.909736 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.909737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.910922 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.910924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.911564 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.911565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.911646 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.911658 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.911729 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.911781 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":44,"thread_start_us":115,"threads_started":1}
I20260927 02:46:06.912046 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.907910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.0.98.2:34565)
0927 02:46:06.908089 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.908096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.908229 (+ 133us) server_negotiation.cc:407] Connection header received
0927 02:46:06.908288 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.908290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.908306 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.908533 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.908724 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.909432 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.910869 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.911535 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.911590 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.911677 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.911925 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.911964 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260927 02:46:06.912053 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.907761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.0.98.3:45085)
0927 02:46:06.908159 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.908163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.908171 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:06.908211 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.908214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.908229 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.908293 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.908470 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.909638 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.910809 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.911521 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.911590 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.911676 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.911925 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.911963 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":86,"threads_started":1}
I20260927 02:46:06.912470 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.907670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45085 (local address 127.0.0.1:53772)
0927 02:46:06.907851 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.907892 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.907966 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.908306 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.908309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.908311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.908403 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.908405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.909736 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.909737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.910732 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.910734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.911713 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.911714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.911780 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.911787 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.911834 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.911882 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":62,"thread_start_us":52,"threads_started":1}
I20260927 02:46:06.912477 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.907672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34565 (local address 127.0.0.1:46290)
0927 02:46:06.908176 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.908183 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.908217 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.908631 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.908632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.908633 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.908668 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.908669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.909511 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.909512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.910793 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.910796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.911714 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.911715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.911780 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.911788 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.911834 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.911882 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260927 02:46:06.913214 19122 tablet_service.cc:1467] Processing CreateTablet for tablet e393610249994e4885846fd9ad2a4a68 (DEFAULT_TABLE table=test-table [id=f7bfc9b05af04b7e8c8807bf331b20f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:06.913347 19253 tablet_service.cc:1467] Processing CreateTablet for tablet e393610249994e4885846fd9ad2a4a68 (DEFAULT_TABLE table=test-table [id=f7bfc9b05af04b7e8c8807bf331b20f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:06.913555 19122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e393610249994e4885846fd9ad2a4a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:06.913669 19253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e393610249994e4885846fd9ad2a4a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:06.916468 19465 tablet_bootstrap.cc:492] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0: Bootstrap starting.
I20260927 02:46:06.916517 19464 tablet_bootstrap.cc:492] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: Bootstrap starting.
I20260927 02:46:06.916757 19384 tablet_service.cc:1467] Processing CreateTablet for tablet e393610249994e4885846fd9ad2a4a68 (DEFAULT_TABLE table=test-table [id=f7bfc9b05af04b7e8c8807bf331b20f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:06.917054 19464 tablet_bootstrap.cc:654] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:06.917100 19384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e393610249994e4885846fd9ad2a4a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:06.917122 19465 tablet_bootstrap.cc:654] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:06.917336 19464 log.cc:824] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:06.917402 19465 log.cc:824] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:06.917971 19464 tablet_bootstrap.cc:492] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: No bootstrap required, opened a new log
I20260927 02:46:06.918030 19464 ts_tablet_manager.cc:1397] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:06.919751 19468 tablet_bootstrap.cc:492] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: Bootstrap starting.
I20260927 02:46:06.920066 19464 raft_consensus.cc:348] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } }
I20260927 02:46:06.920250 19464 raft_consensus.cc:374] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:06.920286 19464 raft_consensus.cc:729] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62cf7de7d9564d84a616aeca58046f08, State: Initialized, Role: FOLLOWER
I20260927 02:46:06.920379 19468 tablet_bootstrap.cc:654] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:06.920436 19464 consensus_queue.cc:260] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } }
I20260927 02:46:06.920698 19464 ts_tablet_manager.cc:1428] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:06.920733 19319 heartbeater.cc:503] Master 127.0.98.62:39637 was elected leader, sending a full tablet report...
I20260927 02:46:06.920761 19468 log.cc:824] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:06.921231 19465 tablet_bootstrap.cc:492] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0: No bootstrap required, opened a new log
I20260927 02:46:06.921351 19465 ts_tablet_manager.cc:1397] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 02:46:06.921393 19468 tablet_bootstrap.cc:492] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: No bootstrap required, opened a new log
I20260927 02:46:06.921440 19468 ts_tablet_manager.cc:1397] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:06.923272 19468 raft_consensus.cc:348] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } }
I20260927 02:46:06.923393 19468 raft_consensus.cc:374] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:06.923414 19468 raft_consensus.cc:729] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81221453becc4bf9ba92bdea0398c74b, State: Initialized, Role: FOLLOWER
I20260927 02:46:06.923358 19465 raft_consensus.cc:348] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } }
I20260927 02:46:06.923523 19465 raft_consensus.cc:374] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:06.923570 19465 raft_consensus.cc:729] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea84889e02b413199a570ee8decbac0, State: Initialized, Role: FOLLOWER
I20260927 02:46:06.923652 19468 consensus_queue.cc:260] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } }
I20260927 02:46:06.923678 19465 consensus_queue.cc:260] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } }
I20260927 02:46:06.923923 19468 ts_tablet_manager.cc:1428] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:46:06.923930 19450 heartbeater.cc:503] Master 127.0.98.62:39637 was elected leader, sending a full tablet report...
I20260927 02:46:06.923980 19465 ts_tablet_manager.cc:1428] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:06.924018 19188 heartbeater.cc:503] Master 127.0.98.62:39637 was elected leader, sending a full tablet report...
I20260927 02:46:06.957912 19469 raft_consensus.cc:482] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:06.958050 19469 raft_consensus.cc:504] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } }
I20260927 02:46:06.958534 19469 leader_election.cc:288] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eea84889e02b413199a570ee8decbac0 (127.0.98.1:45021), 81221453becc4bf9ba92bdea0398c74b (127.0.98.3:45085)
I20260927 02:46:06.962536 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.958758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36239 (local address 127.0.98.1:45021)
0927 02:46:06.958799 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.958801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.958809 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:06.958838 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.958839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.958850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.958901 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.959176 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.959951 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.961541 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.962159 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.962234 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.962243 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.962420 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.962464 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:46:06.962536 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.958826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35811 (local address 127.0.98.3:45085)
0927 02:46:06.958897 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.958900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.959059 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:46:06.959158 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.959159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.959173 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.959239 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:06.959356 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.960027 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.961533 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.962155 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.962226 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.962237 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.962420 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.962463 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:46:06.962559 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.958650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45021 (local address 127.0.98.2:36239)
0927 02:46:06.958715 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.958728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.958768 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.958945 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.958947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.958949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.959017 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.959019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.960030 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.960031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.961439 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.961443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.962243 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.962244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.962332 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.962340 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.962405 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.962456 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":65}
I20260927 02:46:06.962568 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.958750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45085 (local address 127.0.98.2:35811)
0927 02:46:06.958942 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:06.958953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:06.959050 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:06.959245 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:06.959245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:06.959247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:06.959299 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.959300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.960089 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.960090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.961441 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:06.961443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.962243 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.962244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:06.962332 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.962340 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 02:46:06.962405 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:46:06.962452 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":28,"thread_start_us":123,"threads_started":1}
I20260927 02:46:06.962931 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e393610249994e4885846fd9ad2a4a68" candidate_uuid: "62cf7de7d9564d84a616aeca58046f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81221453becc4bf9ba92bdea0398c74b" is_pre_election: true
I20260927 02:46:06.962968 19142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e393610249994e4885846fd9ad2a4a68" candidate_uuid: "62cf7de7d9564d84a616aeca58046f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea84889e02b413199a570ee8decbac0" is_pre_election: true
I20260927 02:46:06.963079 19404 raft_consensus.cc:2454] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62cf7de7d9564d84a616aeca58046f08 in term 0.
I20260927 02:46:06.963193 19142 raft_consensus.cc:2454] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62cf7de7d9564d84a616aeca58046f08 in term 0.
I20260927 02:46:06.963408 19209 leader_election.cc:302] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62cf7de7d9564d84a616aeca58046f08, eea84889e02b413199a570ee8decbac0; no voters:
I20260927 02:46:06.963598 19469 raft_consensus.cc:2781] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:06.963649 19469 raft_consensus.cc:482] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:06.963663 19469 raft_consensus.cc:3037] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:06.964339 19469 raft_consensus.cc:504] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } }
I20260927 02:46:06.964496 19469 leader_election.cc:288] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [CANDIDATE]: Term 1 election: Requested vote from peers eea84889e02b413199a570ee8decbac0 (127.0.98.1:45021), 81221453becc4bf9ba92bdea0398c74b (127.0.98.3:45085)
I20260927 02:46:06.964670 19142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e393610249994e4885846fd9ad2a4a68" candidate_uuid: "62cf7de7d9564d84a616aeca58046f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea84889e02b413199a570ee8decbac0"
I20260927 02:46:06.964668 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e393610249994e4885846fd9ad2a4a68" candidate_uuid: "62cf7de7d9564d84a616aeca58046f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81221453becc4bf9ba92bdea0398c74b"
I20260927 02:46:06.964758 19404 raft_consensus.cc:3037] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:06.964759 19142 raft_consensus.cc:3037] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:06.965682 19404 raft_consensus.cc:2454] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62cf7de7d9564d84a616aeca58046f08 in term 1.
I20260927 02:46:06.965688 19142 raft_consensus.cc:2454] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62cf7de7d9564d84a616aeca58046f08 in term 1.
I20260927 02:46:06.965878 19206 leader_election.cc:302] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62cf7de7d9564d84a616aeca58046f08, 81221453becc4bf9ba92bdea0398c74b; no voters:
I20260927 02:46:06.965968 19469 raft_consensus.cc:2781] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:06.966114 19469 raft_consensus.cc:686] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 1 LEADER]: Becoming Leader. State: Replica: 62cf7de7d9564d84a616aeca58046f08, State: Running, Role: LEADER
I20260927 02:46:06.966212 19469 consensus_queue.cc:237] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } }
I20260927 02:46:06.967119 18999 catalog_manager.cc:5495] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62cf7de7d9564d84a616aeca58046f08 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "62cf7de7d9564d84a616aeca58046f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf7de7d9564d84a616aeca58046f08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 } health_report { overall_health: UNKNOWN } } }
I20260927 02:46:06.986238 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:06.983431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.0.98.2:34565)
0927 02:46:06.983476 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:06.983478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:06.983502 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:46:06.983595 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:06.983596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:06.983617 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:06.983681 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:06.983847 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.984521 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.985302 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:06.985527 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:06.985593 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:06.985808 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:06.985909 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:06.985994 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:06.986143 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:46:06.986154 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:46:06.987970 19404 raft_consensus.cc:1261] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [term 1 FOLLOWER]: Refusing update from remote peer 62cf7de7d9564d84a616aeca58046f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:06.987970 19142 raft_consensus.cc:1261] T e393610249994e4885846fd9ad2a4a68 P eea84889e02b413199a570ee8decbac0 [term 1 FOLLOWER]: Refusing update from remote peer 62cf7de7d9564d84a616aeca58046f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:06.988343 19474 consensus_queue.cc:1035] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81221453becc4bf9ba92bdea0398c74b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:06.988507 19469 consensus_queue.cc:1035] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea84889e02b413199a570ee8decbac0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:06.991533 19478 mvcc.cc:205] Tried to move back new op lower bound from 7333794475980316672 to 7333794475894358016. Current Snapshot: MvccSnapshot[applied={T|T < 7333794475980316672 or (T in {7333794475980316672})}]
I20260927 02:46:06.993705 19000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:06.993801 19000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:06.994460 19000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02cdd407068640a68fba3264011a72a8: Sending DeleteTablet for 3 replicas of tablet e393610249994e4885846fd9ad2a4a68
I20260927 02:46:06.994783 19253 tablet_service.cc:1514] Processing DeleteTablet for tablet e393610249994e4885846fd9ad2a4a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:06 UTC) from {username='slave'} at 127.0.0.1:46290
I20260927 02:46:06.994830 19384 tablet_service.cc:1514] Processing DeleteTablet for tablet e393610249994e4885846fd9ad2a4a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:06 UTC) from {username='slave'} at 127.0.0.1:53772
I20260927 02:46:06.994838 19122 tablet_service.cc:1514] Processing DeleteTablet for tablet e393610249994e4885846fd9ad2a4a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:06 UTC) from {username='slave'} at 127.0.0.1:45856
I20260927 02:46:06.995188 19487 tablet_replica.cc:330] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: stopping tablet replica
I20260927 02:46:06.995291 19487 raft_consensus.cc:2229] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:46:06.995416 19487 raft_consensus.cc:2258] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:06.995445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 19060
I20260927 02:46:06.995563 19488 tablet_replica.cc:330] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: stopping tablet replica
I20260927 02:46:06.995642 19488 raft_consensus.cc:2229] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:06.995715 19488 raft_consensus.cc:2258] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:06.996347 19487 ts_tablet_manager.cc:1905] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:06.996450 19488 ts_tablet_manager.cc:1905] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:06.998114 19488 ts_tablet_manager.cc:1918] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:06.998107 19487 ts_tablet_manager.cc:1918] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:06.998178 19488 log.cc:1196] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/e393610249994e4885846fd9ad2a4a68
I20260927 02:46:06.998282 19487 log.cc:1196] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/e393610249994e4885846fd9ad2a4a68
I20260927 02:46:06.998417 19488 ts_tablet_manager.cc:1939] T e393610249994e4885846fd9ad2a4a68 P 81221453becc4bf9ba92bdea0398c74b: Deleting consensus metadata
I20260927 02:46:06.998648 19487 ts_tablet_manager.cc:1939] T e393610249994e4885846fd9ad2a4a68 P 62cf7de7d9564d84a616aeca58046f08: Deleting consensus metadata
I20260927 02:46:06.998893 18984 catalog_manager.cc:4841] TS 81221453becc4bf9ba92bdea0398c74b (127.0.98.3:45085): tablet e393610249994e4885846fd9ad2a4a68 (table test-table [id=f7bfc9b05af04b7e8c8807bf331b20f3]) successfully deleted
I20260927 02:46:06.999024 18985 catalog_manager.cc:4841] TS 62cf7de7d9564d84a616aeca58046f08 (127.0.98.2:34565): tablet e393610249994e4885846fd9ad2a4a68 (table test-table [id=f7bfc9b05af04b7e8c8807bf331b20f3]) successfully deleted
W20260927 02:46:07.002986 18987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45021 (error 108)
I20260927 02:46:07.003481 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:07.003350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45021 (local address 127.0.0.1:45862)
0927 02:46:07.003395 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:07.003443 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:46:07.003535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 19191
W20260927 02:46:07.004995 18987 catalog_manager.cc:4584] TS eea84889e02b413199a570ee8decbac0 (127.0.98.1:45021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e393610249994e4885846fd9ad2a4a68: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45021: connect: Connection refused (error 111)
I20260927 02:46:07.007759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 19322
I20260927 02:46:07.011545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 18969
2026-09-27T02:46:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (685 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:07 dist-test-slave-44kr krb5kdc[19493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:07 dist-test-slave-44kr krb5kdc[19493](info): set up 2 sockets
Sep 27 02:46:07 dist-test-slave-44kr krb5kdc[19493](info): commencing operation
krb5kdc: starting...
W20260927 02:46:07.963994 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:08 dist-test-slave-44kr krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:08.084935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37715
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:08.166572 19506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:08.166792 19506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:08.166823 19506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:08.167994 19506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:08.168042 19506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:08.168061 19506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:08.168078 19506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:08.168092 19506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:08.169636 19506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:08.169880 19506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:08.170091 19506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:08.174876 19514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:08.174844 19511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:08.174852 19512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:08.174983 19506 server_base.cc:1034] running on GCE node
I20260927 02:46:08.175285 19506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:08.175603 19506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:08.176738 19506 hybrid_clock.cc:648] HybridClock initialized: now 1790477168176730 us; error 30 us; skew 500 ppm
Sep 27 02:46:08 dist-test-slave-44kr krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:08.179163 19506 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:08.179612 19506 webserver.cc:460] Webserver started at http://127.0.98.62:33673/ using document root <none> and password file <none>
I20260927 02:46:08.179783 19506 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:08.179822 19506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:08.179906 19506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:08.180974 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "59ff5fc5744f4285831acf79030f64c5"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "225284e8bba36473211cb399197c4b38"
server_key_iv: "b74004068e46aa24fdeb71696c7c4196"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.181272 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "59ff5fc5744f4285831acf79030f64c5"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "225284e8bba36473211cb399197c4b38"
server_key_iv: "b74004068e46aa24fdeb71696c7c4196"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.182510 19506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:46:08.183248 19521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.183459 19506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:08.183547 19506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "59ff5fc5744f4285831acf79030f64c5"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "225284e8bba36473211cb399197c4b38"
server_key_iv: "b74004068e46aa24fdeb71696c7c4196"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.183614 19506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:08.211565 19506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:08.213850 19506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:08.213990 19506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:08.217784 19506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37715
I20260927 02:46:08.217819 19573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37715 every 8 connection(s)
I20260927 02:46:08.218149 19506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:08.218964 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:08.219913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 19506
I20260927 02:46:08.220036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:08.220252 392 external_mini_cluster.cc:1428] Setting key 0878aec291894e590b3699b333566112
I20260927 02:46:08.221807 19574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5: Bootstrap starting.
I20260927 02:46:08.222586 19574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:08.222859 19574 log.cc:824] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:08.223532 19574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5: No bootstrap required, opened a new log
I20260927 02:46:08.225841 19574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff5fc5744f4285831acf79030f64c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37715 } }
I20260927 02:46:08.226037 19574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:08.226075 19574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ff5fc5744f4285831acf79030f64c5, State: Initialized, Role: FOLLOWER
I20260927 02:46:08.226215 19574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff5fc5744f4285831acf79030f64c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37715 } }
I20260927 02:46:08.226298 19574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:08.226338 19574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:08.226385 19574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:08.227123 19574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff5fc5744f4285831acf79030f64c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37715 } }
I20260927 02:46:08.227229 19574 leader_election.cc:302] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59ff5fc5744f4285831acf79030f64c5; no voters:
I20260927 02:46:08.227406 19574 leader_election.cc:288] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 27 02:46:08 dist-test-slave-44kr krb5kdc[19493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:08.227491 19579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:08.227663 19574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:08.227722 19579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [term 1 LEADER]: Becoming Leader. State: Replica: 59ff5fc5744f4285831acf79030f64c5, State: Running, Role: LEADER
I20260927 02:46:08.227852 19579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff5fc5744f4285831acf79030f64c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37715 } }
I20260927 02:46:08.228422 19579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59ff5fc5744f4285831acf79030f64c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff5fc5744f4285831acf79030f64c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37715 } } }
W20260927 02:46:08.229359 19593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:08.229403 19593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:08.228497 19581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59ff5fc5744f4285831acf79030f64c5. Latest consensus state: current_term: 1 leader_uuid: "59ff5fc5744f4285831acf79030f64c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff5fc5744f4285831acf79030f64c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37715 } } }
I20260927 02:46:08.229840 19581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:08.228516 19579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:08.230019 19596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:08.230172 19596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:08.231609 19596 catalog_manager.cc:1347] Generated new cluster ID: 10937e700aeb4252ac4efb7ec66553bc
I20260927 02:46:08.231659 19596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:08.233424 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.220905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.0.98.62:37715)
0927 02:46:08.221076 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:08.221085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:08.221117 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:46:08.221570 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:08.221577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:08.221630 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:08.221762 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:08.222577 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.223619 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.224696 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.225171 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.225263 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.228025 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:08.228030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:08.228030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:08.228044 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:08.231102 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.231603 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.231606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.231607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.231758 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.232005 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.232006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.232006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.232319 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:08.232505 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:08.233062 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 02:46:08.233205 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260927 02:46:08.244690 19596 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:08.245229 19596 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:08.248374 19596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5: Generated new TSK 0
I20260927 02:46:08.248597 19596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:46:08.279814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37715
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:08.376325 19600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:08.376578 19600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:08.376602 19600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:08.377789 19600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:08.377833 19600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:08.377858 19600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:08.379386 19600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:08.379619 19600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:08.379839 19600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:08.384740 19606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:08.384785 19605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:08.384814 19608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:08.385185 19600 server_base.cc:1034] running on GCE node
I20260927 02:46:08.385401 19600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:08.385666 19600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:08.386842 19600 hybrid_clock.cc:648] HybridClock initialized: now 1790477168386825 us; error 35 us; skew 500 ppm
Sep 27 02:46:08 dist-test-slave-44kr krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:08.389766 19600 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:46:08.390240 19600 webserver.cc:460] Webserver started at http://127.0.98.1:39157/ using document root <none> and password file <none>
I20260927 02:46:08.390443 19600 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:08.390496 19600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:08.390601 19600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:08.391625 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "542440ff98ff4c559dc0bdf82da2909b"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "8677cddc4ba35f2d18ebc3713c77185f"
server_key_iv: "6b8cde92c2aac4493ef5f9346befcab6"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.391938 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "542440ff98ff4c559dc0bdf82da2909b"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "8677cddc4ba35f2d18ebc3713c77185f"
server_key_iv: "6b8cde92c2aac4493ef5f9346befcab6"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.393262 19600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:46:08.394026 19615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.394263 19600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:08.394352 19600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "542440ff98ff4c559dc0bdf82da2909b"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "8677cddc4ba35f2d18ebc3713c77185f"
server_key_iv: "6b8cde92c2aac4493ef5f9346befcab6"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.394418 19600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:08.415453 19600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:08.417804 19600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:08.417958 19600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:08.418192 19600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:08.418499 19600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:08.418532 19600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.418562 19600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:08.418579 19600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.424235 19600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41923
I20260927 02:46:08.424369 19728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41923 every 8 connection(s)
I20260927 02:46:08.424638 19600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:08.425348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 19600
I20260927 02:46:08.425464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:46:08.425683 392 external_mini_cluster.cc:1428] Setting key ac5de7f66189750732c1e95b165d3275
Sep 27 02:46:08 dist-test-slave-44kr krb5kdc[19493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:46:08.434588 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.425392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46129 (local address 127.0.98.62:37715)
0927 02:46:08.425454 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:08.425456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:08.426010 (+ 554us) server_negotiation.cc:407] Connection header received
0927 02:46:08.427128 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:08.427129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:08.427140 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:08.427189 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:08.427873 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.428406 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.429467 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.429739 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.429798 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.432055 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:08.432057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:08.432058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:08.432065 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:08.433370 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.433820 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.433821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.433822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.433870 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.434050 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.434050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.434050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.434144 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:08.434214 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:08.434448 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 02:46:08.434518 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:46:08.434760 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.425332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37715 (local address 127.0.98.1:46129)
0927 02:46:08.425881 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:08.425920 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:08.427018 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:08.427264 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:08.427269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:08.427523 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:08.427724 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.427728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.428499 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.428500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.429379 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.429381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.429813 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.429813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.429927 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.430450 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:08.430457 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:08.431960 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:08.433488 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:08.433496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:08.433499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:08.433760 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:08.433919 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:08.433919 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:08.433919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:08.434009 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:08.434190 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:08.434194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:08.434341 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 02:46:08.434424 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:46:08.434543 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":74,"threads_started":1}
I20260927 02:46:08.435194 19729 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37715
I20260927 02:46:08.435282 19729 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:08.435420 19729 heartbeater.cc:511] Master 127.0.98.62:37715 requested a full tablet report, sending...
I20260927 02:46:08.435889 19538 ts_manager.cc:194] Registered new tserver with Master: 542440ff98ff4c559dc0bdf82da2909b (127.0.98.1:41923)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:46:08.436695 19538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:46:08.451959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37715
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:08.531025 19734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:08.531230 19734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:08.531253 19734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:08.532436 19734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:08.532487 19734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:08.532513 19734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:08.534017 19734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:08.534252 19734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:08.534461 19734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:08.539354 19742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:08.539400 19739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:08.539533 19734 server_base.cc:1034] running on GCE node
W20260927 02:46:08.539373 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:08.539774 19734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:08.540006 19734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:08.541184 19734 hybrid_clock.cc:648] HybridClock initialized: now 1790477168541166 us; error 28 us; skew 500 ppm
Sep 27 02:46:08 dist-test-slave-44kr krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:08.543778 19734 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:46:08.544224 19734 webserver.cc:460] Webserver started at http://127.0.98.2:42451/ using document root <none> and password file <none>
I20260927 02:46:08.544472 19734 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:08.544526 19734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:08.544657 19734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:08.545717 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "4de4d388c106ef24cb2da4784aa03313"
server_key_iv: "60b7678275a65847b301e1309ba9183b"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.546041 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "4de4d388c106ef24cb2da4784aa03313"
server_key_iv: "60b7678275a65847b301e1309ba9183b"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.547298 19734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:46:08.548028 19749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.548223 19734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:08.548305 19734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "4de4d388c106ef24cb2da4784aa03313"
server_key_iv: "60b7678275a65847b301e1309ba9183b"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.548363 19734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:08.573907 19734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:08.576062 19734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:08.576207 19734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:08.576449 19734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:08.576769 19734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:08.576804 19734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.576834 19734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:08.576857 19734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.582417 19734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44041
I20260927 02:46:08.582516 19862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44041 every 8 connection(s)
I20260927 02:46:08.582798 19734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:08.587033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 19734
I20260927 02:46:08.587152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:46:08.587399 392 external_mini_cluster.cc:1428] Setting key 67cef9a2eb2cc50ee1078e52608a1939
Sep 27 02:46:08 dist-test-slave-44kr krb5kdc[19493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:08.592821 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.583470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44441 (local address 127.0.98.62:37715)
0927 02:46:08.583536 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:08.583538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:08.583704 (+ 166us) server_negotiation.cc:407] Connection header received
0927 02:46:08.584670 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:08.584672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:08.584684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:08.584758 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:08.585379 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.585910 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.587068 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.587385 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.587496 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.589848 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:08.589850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:08.589850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:08.589858 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:08.591348 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.591815 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.591816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.591816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.591875 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.592134 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.592135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.592135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.592273 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:08.592345 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:08.592639 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 02:46:08.592724 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:46:08.592897 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.583441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37715 (local address 127.0.98.2:44441)
0927 02:46:08.583589 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:08.583630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:08.584558 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:08.584813 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:08.584816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:08.585063 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:08.585262 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.585266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.585984 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.585985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.586976 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.586979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.587552 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.587553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.587681 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.588209 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:08.588219 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:08.589759 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:08.591472 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:08.591478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:08.591480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:08.591743 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:08.591965 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:08.591966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:08.591966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:08.592071 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:08.592340 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:08.592345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:08.592508 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 02:46:08.592597 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:46:08.592721 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260927 02:46:08.593528 19863 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37715
I20260927 02:46:08.593623 19863 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:08.593806 19863 heartbeater.cc:511] Master 127.0.98.62:37715 requested a full tablet report, sending...
I20260927 02:46:08.594192 19538 ts_manager.cc:194] Registered new tserver with Master: 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 (127.0.98.2:44041)
I20260927 02:46:08.594570 19538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44441
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:46:08.613372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37715
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:08.694270 19868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:08.694486 19868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:08.694517 19868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:08.695705 19868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:08.695750 19868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:08.695784 19868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:08.697326 19868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:08.697561 19868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:08.697781 19868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:08.702843 19873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:08.702845 19874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:08.702838 19876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:08.702860 19868 server_base.cc:1034] running on GCE node
I20260927 02:46:08.703334 19868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:08.703609 19868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:08.704778 19868 hybrid_clock.cc:648] HybridClock initialized: now 1790477168704761 us; error 33 us; skew 500 ppm
Sep 27 02:46:08 dist-test-slave-44kr krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:08.707372 19868 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:46:08.707765 19868 webserver.cc:460] Webserver started at http://127.0.98.3:32895/ using document root <none> and password file <none>
I20260927 02:46:08.707958 19868 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:08.707998 19868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:08.708089 19868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:08.709131 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "a6ce22668bb5496382e305b96a5cb76c"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "abc3f95f22baf04b584f3a9ee6bea473"
server_key_iv: "40a96ffef45459bc3871ec1f43931bdf"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.709417 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a6ce22668bb5496382e305b96a5cb76c"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "abc3f95f22baf04b584f3a9ee6bea473"
server_key_iv: "40a96ffef45459bc3871ec1f43931bdf"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.710662 19868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:46:08.711441 19883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.711709 19868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:08.711792 19868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "a6ce22668bb5496382e305b96a5cb76c"
format_stamp: "Formatted at 2026-09-27 02:46:08 on dist-test-slave-44kr"
server_key: "abc3f95f22baf04b584f3a9ee6bea473"
server_key_iv: "40a96ffef45459bc3871ec1f43931bdf"
server_key_version: "encryptionkey@0"
I20260927 02:46:08.711865 19868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:08.724869 19868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:08.726950 19868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:08.727088 19868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:08.727299 19868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:08.727625 19868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:08.727667 19868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.727696 19868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:08.727720 19868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:08.733265 19868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43301
I20260927 02:46:08.733314 19996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43301 every 8 connection(s)
I20260927 02:46:08.733601 19868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:08.737890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 19868
I20260927 02:46:08.738009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:46:08.738204 392 external_mini_cluster.cc:1428] Setting key 81e9d3750890da61726510b4cc948e59
Sep 27 02:46:08 dist-test-slave-44kr krb5kdc[19493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:08.742801 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.734258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59817 (local address 127.0.98.62:37715)
0927 02:46:08.734306 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:08.734307 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:08.734496 (+ 189us) server_negotiation.cc:407] Connection header received
0927 02:46:08.735378 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:08.735379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:08.735395 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:08.735457 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:08.736082 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.736667 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.737740 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.737944 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.738112 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.740475 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:08.740476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:08.740477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:08.740484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:08.741628 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.742022 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.742022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.742022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.742070 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.742251 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.742252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.742252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.742351 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:08.742395 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:08.742675 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 02:46:08.742733 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:46:08.743031 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.734229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37715 (local address 127.0.98.3:59817)
0927 02:46:08.734372 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:08.734414 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:08.735291 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:08.735523 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:08.735541 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:08.735793 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:08.735980 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.735984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.736760 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.736761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.737622 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.737625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.738048 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.738048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.738206 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.738820 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:08.738840 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:08.740356 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:08.741701 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:08.741707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:08.741709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:08.741922 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:08.742115 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:08.742116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:08.742116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:08.742211 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:08.742421 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:08.742427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:08.742562 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 02:46:08.742658 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 02:46:08.742810 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260927 02:46:08.743500 19997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37715
I20260927 02:46:08.743590 19997 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:08.743759 19997 heartbeater.cc:511] Master 127.0.98.62:37715 requested a full tablet report, sending...
I20260927 02:46:08.744102 19538 ts_manager.cc:194] Registered new tserver with Master: a6ce22668bb5496382e305b96a5cb76c (127.0.98.3:43301)
I20260927 02:46:08.744513 19538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59817
I20260927 02:46:08.750036 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:08.757077 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.751253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.0.98.62:37715)
0927 02:46:08.751307 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:08.751309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:08.751384 (+ 75us) server_negotiation.cc:407] Connection header received
0927 02:46:08.751482 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:08.751483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:08.751493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:08.751541 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:08.752156 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.752712 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.753457 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.753661 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.753715 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.754848 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:08.754850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:08.754851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:08.754858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:08.756002 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.756309 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.756310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.756310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.756362 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:08.756565 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:08.756566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:08.756566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:08.756713 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:08.756800 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:08.756909 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 02:46:08.756984 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:46:08.758598 19538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:08.760100 19538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:08.766914 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.762762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.0.98.2:44041)
0927 02:46:08.762939 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:08.762945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:08.762958 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:46:08.763000 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:08.763003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:08.763018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:08.763162 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:08.763321 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.764478 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.765829 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.766547 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.766624 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.766688 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:08.766785 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 02:46:08.766825 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260927 02:46:08.766912 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.762642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44041 (local address 127.0.0.1:35574)
0927 02:46:08.762797 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:08.762841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:08.762905 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:08.763166 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:08.763170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:08.763172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:08.763250 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.763253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.764571 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.764572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.765752 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.765754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.766626 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.766626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.766693 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.766704 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 02:46:08.766773 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:46:08.766828 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":112,"thread_start_us":64,"threads_started":1}
I20260927 02:46:08.767318 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.762646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43301 (local address 127.0.0.1:59570)
0927 02:46:08.762960 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:08.762969 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:08.763005 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:08.763429 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:08.763430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:08.763432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:08.763477 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.763478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.764672 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.764673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.765938 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.765939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.766742 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.766743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.766805 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.766808 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:08.766856 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:46:08.766889 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":65,"thread_start_us":99,"threads_started":1}
I20260927 02:46:08.767565 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.762823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.0.98.3:43301)
0927 02:46:08.763086 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:08.763091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:08.763100 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:46:08.763136 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:08.763138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:08.763153 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:08.763218 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:08.763543 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.764581 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.766013 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.766674 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.766736 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.766785 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:08.766956 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 02:46:08.767488 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":72,"threads_started":1}
I20260927 02:46:08.767791 19797 tablet_service.cc:1467] Processing CreateTablet for tablet d58f42c9e05a450592b9a715e85621dc (DEFAULT_TABLE table=test-table [id=3258b64b4697489a9bce30ecbc5865d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:08.768139 19797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58f42c9e05a450592b9a715e85621dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:08.768203 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.762848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41923 (local address 127.0.0.1:41906)
0927 02:46:08.763313 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:08.763323 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:08.763376 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:08.763642 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:08.763643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:08.763645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:08.763681 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.763683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.764789 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.764790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.765931 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:08.765934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.767910 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.767911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:08.768023 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.768027 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:08.768090 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:46:08.768132 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
I20260927 02:46:08.768240 19931 tablet_service.cc:1467] Processing CreateTablet for tablet d58f42c9e05a450592b9a715e85621dc (DEFAULT_TABLE table=test-table [id=3258b64b4697489a9bce30ecbc5865d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:08.768507 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:08.763065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.0.98.1:41923)
0927 02:46:08.763247 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:08.763251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:08.763423 (+ 172us) server_negotiation.cc:407] Connection header received
0927 02:46:08.763465 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:08.763467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:08.763483 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:08.763553 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:08.763735 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.764488 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.766821 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:08.767546 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:08.767625 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:08.767681 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:08.768359 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 02:46:08.768434 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260927 02:46:08.768594 19931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58f42c9e05a450592b9a715e85621dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:08.769122 19663 tablet_service.cc:1467] Processing CreateTablet for tablet d58f42c9e05a450592b9a715e85621dc (DEFAULT_TABLE table=test-table [id=3258b64b4697489a9bce30ecbc5865d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:08.769428 19663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58f42c9e05a450592b9a715e85621dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:08.771155 20014 tablet_bootstrap.cc:492] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: Bootstrap starting.
I20260927 02:46:08.771653 20015 tablet_bootstrap.cc:492] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: Bootstrap starting.
I20260927 02:46:08.771919 20014 tablet_bootstrap.cc:654] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:08.772367 20015 tablet_bootstrap.cc:654] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:08.772538 20016 tablet_bootstrap.cc:492] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b: Bootstrap starting.
I20260927 02:46:08.772737 20014 log.cc:824] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:08.772962 20015 log.cc:824] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:08.773167 20016 tablet_bootstrap.cc:654] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:08.773521 20014 tablet_bootstrap.cc:492] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: No bootstrap required, opened a new log
I20260927 02:46:08.773609 20014 ts_tablet_manager.cc:1397] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 02:46:08.773631 20015 tablet_bootstrap.cc:492] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: No bootstrap required, opened a new log
I20260927 02:46:08.773687 20015 ts_tablet_manager.cc:1397] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:46:08.773705 20016 log.cc:824] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:08.774416 20016 tablet_bootstrap.cc:492] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b: No bootstrap required, opened a new log
I20260927 02:46:08.774484 20016 ts_tablet_manager.cc:1397] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:08.775444 20015 raft_consensus.cc:348] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:08.775449 20014 raft_consensus.cc:348] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:08.775588 20015 raft_consensus.cc:374] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:08.775619 20015 raft_consensus.cc:729] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ab4e1e98cf45c38cdf0fa3cb5ab9f7, State: Initialized, Role: FOLLOWER
I20260927 02:46:08.775619 20014 raft_consensus.cc:374] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:08.775653 20014 raft_consensus.cc:729] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ce22668bb5496382e305b96a5cb76c, State: Initialized, Role: FOLLOWER
I20260927 02:46:08.775749 20015 consensus_queue.cc:260] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:08.775779 20014 consensus_queue.cc:260] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:08.776037 20014 ts_tablet_manager.cc:1428] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 02:46:08.776050 20015 ts_tablet_manager.cc:1428] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:46:08.776067 19997 heartbeater.cc:503] Master 127.0.98.62:37715 was elected leader, sending a full tablet report...
I20260927 02:46:08.776309 19863 heartbeater.cc:503] Master 127.0.98.62:37715 was elected leader, sending a full tablet report...
I20260927 02:46:08.776969 20016 raft_consensus.cc:348] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:08.777148 20016 raft_consensus.cc:374] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:08.777190 20016 raft_consensus.cc:729] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542440ff98ff4c559dc0bdf82da2909b, State: Initialized, Role: FOLLOWER
I20260927 02:46:08.777295 20016 consensus_queue.cc:260] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:08.777515 20016 ts_tablet_manager.cc:1428] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:46:08.777560 19729 heartbeater.cc:503] Master 127.0.98.62:37715 was elected leader, sending a full tablet report...
W20260927 02:46:08.833323 19864 tablet.cc:2369] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:46:08.925390 19730 tablet.cc:2369] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:46:08.984135 19998 tablet.cc:2369] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:09.143784 20022 raft_consensus.cc:482] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:09.143913 20022 raft_consensus.cc:504] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:09.144299 20022 leader_election.cc:288] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 (127.0.98.2:44041), a6ce22668bb5496382e305b96a5cb76c (127.0.98.3:43301)
I20260927 02:46:09.146669 20020 raft_consensus.cc:482] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:09.146801 20020 raft_consensus.cc:504] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:09.147173 20020 leader_election.cc:288] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 (127.0.98.2:44041), 542440ff98ff4c559dc0bdf82da2909b (127.0.98.1:41923)
I20260927 02:46:09.147333 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.144577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44783 (local address 127.0.98.2:44041)
0927 02:46:09.144625 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.144627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.144756 (+ 129us) server_negotiation.cc:407] Connection header received
0927 02:46:09.144792 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.144793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.144805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.144854 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:09.145043 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.145531 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.146530 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.146981 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.147045 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.147056 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.147217 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.147257 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:46:09.147403 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.144636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42289 (local address 127.0.98.3:43301)
0927 02:46:09.144707 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.144709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.144722 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:46:09.144801 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.144802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.144811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.144861 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:09.145018 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.145514 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.146537 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.147072 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.147152 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.147162 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.147309 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.147350 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:46:09.147540 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.144508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44041 (local address 127.0.98.1:44783)
0927 02:46:09.144664 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.144675 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.144740 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.144868 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.144868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.144871 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:09.144979 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.144980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.145595 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.145596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.146433 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.146436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.147056 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.147057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.147130 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.147134 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.147413 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.147457 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":17,"thread_start_us":83,"threads_started":1}
I20260927 02:46:09.147709 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.144521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43301 (local address 127.0.98.1:42289)
0927 02:46:09.144671 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.144681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.144740 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.144871 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.144872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.144874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:09.144963 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.144965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.145599 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.145600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.146450 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.146453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.147160 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.147161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.147230 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.147232 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.147504 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.147537 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":49,"thread_start_us":83,"threads_started":1}
I20260927 02:46:09.148017 19817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f42c9e05a450592b9a715e85621dc" candidate_uuid: "542440ff98ff4c559dc0bdf82da2909b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" is_pre_election: true
I20260927 02:46:09.148182 19817 raft_consensus.cc:2454] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 542440ff98ff4c559dc0bdf82da2909b in term 0.
I20260927 02:46:09.148326 19951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f42c9e05a450592b9a715e85621dc" candidate_uuid: "542440ff98ff4c559dc0bdf82da2909b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ce22668bb5496382e305b96a5cb76c" is_pre_election: true
I20260927 02:46:09.148443 19951 raft_consensus.cc:2454] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 542440ff98ff4c559dc0bdf82da2909b in term 0.
I20260927 02:46:09.148422 19618 leader_election.cc:302] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542440ff98ff4c559dc0bdf82da2909b, 60ab4e1e98cf45c38cdf0fa3cb5ab9f7; no voters:
I20260927 02:46:09.148664 20022 raft_consensus.cc:2781] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:09.148725 20022 raft_consensus.cc:482] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:09.148746 20022 raft_consensus.cc:3037] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:09.149605 20022 raft_consensus.cc:504] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:09.149920 19817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f42c9e05a450592b9a715e85621dc" candidate_uuid: "542440ff98ff4c559dc0bdf82da2909b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7"
I20260927 02:46:09.149925 19951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f42c9e05a450592b9a715e85621dc" candidate_uuid: "542440ff98ff4c559dc0bdf82da2909b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ce22668bb5496382e305b96a5cb76c"
I20260927 02:46:09.150005 19951 raft_consensus.cc:3037] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:09.150002 19817 raft_consensus.cc:3037] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:09.150315 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.147309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44041 (local address 127.0.98.3:40435)
0927 02:46:09.147368 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.147379 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.147414 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.147896 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.147897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.147898 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:09.147959 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.147960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.148717 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.148717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.149468 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.149470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.150090 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.150091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.150159 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.150162 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.150207 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.150251 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":16}
I20260927 02:46:09.150532 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.147613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40435 (local address 127.0.98.2:44041)
0927 02:46:09.147771 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.147772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.147780 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:09.147799 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.147800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.147809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.147857 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:09.148022 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.148658 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.149556 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.149993 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.150051 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.150061 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.150447 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.150481 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 02:46:09.150687 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.147603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57845 (local address 127.0.98.1:41923)
0927 02:46:09.147642 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.147644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.147652 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:09.147735 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.147736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.147746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.147773 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:09.147963 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.148587 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.149831 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.150296 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.150361 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.150370 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.150596 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.150632 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:46:09.150705 19816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f42c9e05a450592b9a715e85621dc" candidate_uuid: "a6ce22668bb5496382e305b96a5cb76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" is_pre_election: true
I20260927 02:46:09.150820 19951 raft_consensus.cc:2454] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 542440ff98ff4c559dc0bdf82da2909b in term 1.
I20260927 02:46:09.150921 19817 raft_consensus.cc:2454] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 542440ff98ff4c559dc0bdf82da2909b in term 1.
I20260927 02:46:09.150908 20022 leader_election.cc:288] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [CANDIDATE]: Term 1 election: Requested vote from peers 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 (127.0.98.2:44041), a6ce22668bb5496382e305b96a5cb76c (127.0.98.3:43301)
I20260927 02:46:09.151010 19616 leader_election.cc:302] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542440ff98ff4c559dc0bdf82da2909b, a6ce22668bb5496382e305b96a5cb76c; no voters:
I20260927 02:46:09.151151 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.147406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41923 (local address 127.0.98.3:57845)
0927 02:46:09.147606 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.147615 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.147664 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.147828 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.147829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.147830 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:09.147891 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.147892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.148675 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.148676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.149745 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.149748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.150438 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.150438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.150517 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.150520 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.150775 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.150813 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":30,"thread_start_us":109,"threads_started":1}
I20260927 02:46:09.151137 20022 raft_consensus.cc:2781] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:09.151304 20022 raft_consensus.cc:686] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [term 1 LEADER]: Becoming Leader. State: Replica: 542440ff98ff4c559dc0bdf82da2909b, State: Running, Role: LEADER
I20260927 02:46:09.151409 20022 consensus_queue.cc:237] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } }
I20260927 02:46:09.151489 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f42c9e05a450592b9a715e85621dc" candidate_uuid: "a6ce22668bb5496382e305b96a5cb76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542440ff98ff4c559dc0bdf82da2909b" is_pre_election: true
I20260927 02:46:09.151857 19886 leader_election.cc:302] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6ce22668bb5496382e305b96a5cb76c; no voters: 542440ff98ff4c559dc0bdf82da2909b, 60ab4e1e98cf45c38cdf0fa3cb5ab9f7
I20260927 02:46:09.152160 20020 raft_consensus.cc:2726] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 02:46:09.152334 19538 catalog_manager.cc:5495] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b reported cstate change: term changed from 0 to 1, leader changed from <none> to 542440ff98ff4c559dc0bdf82da2909b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "542440ff98ff4c559dc0bdf82da2909b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "542440ff98ff4c559dc0bdf82da2909b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 } health_report { overall_health: UNKNOWN } } }
I20260927 02:46:09.208441 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.206037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.0.98.1:41923)
0927 02:46:09.206095 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.206097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.206104 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:46:09.206139 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.206140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.206158 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.206226 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:09.206350 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.206839 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.207609 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.207803 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.207866 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.208060 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:09.208173 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:09.208235 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.208344 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.208364 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:46:09.210328 19817 raft_consensus.cc:1261] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [term 1 FOLLOWER]: Refusing update from remote peer 542440ff98ff4c559dc0bdf82da2909b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:09.210371 19951 raft_consensus.cc:1261] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 1 FOLLOWER]: Refusing update from remote peer 542440ff98ff4c559dc0bdf82da2909b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:09.210616 20022 consensus_queue.cc:1035] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ab4e1e98cf45c38cdf0fa3cb5ab9f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:09.210755 20026 consensus_queue.cc:1035] T d58f42c9e05a450592b9a715e85621dc P 542440ff98ff4c559dc0bdf82da2909b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6ce22668bb5496382e305b96a5cb76c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:09.213397 20031 mvcc.cc:205] Tried to move back new op lower bound from 7333794485082337280 to 7333794484845117440. Current Snapshot: MvccSnapshot[applied={T|T < 7333794485082337280}]
I20260927 02:46:09.215760 19537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:09.215871 19537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:09.216743 19537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59ff5fc5744f4285831acf79030f64c5: Sending DeleteTablet for 3 replicas of tablet d58f42c9e05a450592b9a715e85621dc
I20260927 02:46:09.217087 19931 tablet_service.cc:1514] Processing DeleteTablet for tablet d58f42c9e05a450592b9a715e85621dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59570
I20260927 02:46:09.217195 19797 tablet_service.cc:1514] Processing DeleteTablet for tablet d58f42c9e05a450592b9a715e85621dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35574
I20260927 02:46:09.217424 20039 tablet_replica.cc:330] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: stopping tablet replica
I20260927 02:46:09.217410 19663 tablet_service.cc:1514] Processing DeleteTablet for tablet d58f42c9e05a450592b9a715e85621dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41906
I20260927 02:46:09.217521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 19600
I20260927 02:46:09.217561 20039 raft_consensus.cc:2229] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:09.219161 20038 tablet_replica.cc:330] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: stopping tablet replica
I20260927 02:46:09.219280 20038 raft_consensus.cc:2229] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:09.219386 20038 raft_consensus.cc:2258] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:09.219813 20038 ts_tablet_manager.cc:1905] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:09.221210 20039 raft_consensus.cc:2258] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:09.221485 20038 ts_tablet_manager.cc:1918] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:09.221557 20038 log.cc:1196] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/d58f42c9e05a450592b9a715e85621dc
I20260927 02:46:09.221812 20038 ts_tablet_manager.cc:1939] T d58f42c9e05a450592b9a715e85621dc P a6ce22668bb5496382e305b96a5cb76c: Deleting consensus metadata
I20260927 02:46:09.222294 19522 catalog_manager.cc:4841] TS a6ce22668bb5496382e305b96a5cb76c (127.0.98.3:43301): tablet d58f42c9e05a450592b9a715e85621dc (table test-table [id=3258b64b4697489a9bce30ecbc5865d3]) successfully deleted
I20260927 02:46:09.222390 20039 ts_tablet_manager.cc:1905] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:09.223891 20039 ts_tablet_manager.cc:1918] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:09.223963 20039 log.cc:1196] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/d58f42c9e05a450592b9a715e85621dc
I20260927 02:46:09.224169 20039 ts_tablet_manager.cc:1939] T d58f42c9e05a450592b9a715e85621dc P 60ab4e1e98cf45c38cdf0fa3cb5ab9f7: Deleting consensus metadata
I20260927 02:46:09.224691 19524 catalog_manager.cc:4841] TS 60ab4e1e98cf45c38cdf0fa3cb5ab9f7 (127.0.98.2:44041): tablet d58f42c9e05a450592b9a715e85621dc (table test-table [id=3258b64b4697489a9bce30ecbc5865d3]) successfully deleted
W20260927 02:46:09.224970 19524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41923 (error 108)
I20260927 02:46:09.225455 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.225344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41923 (local address 127.0.0.1:41930)
0927 02:46:09.225390 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.225425 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:46:09.225502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 19734
W20260927 02:46:09.225584 19524 catalog_manager.cc:4584] TS 542440ff98ff4c559dc0bdf82da2909b (127.0.98.1:41923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d58f42c9e05a450592b9a715e85621dc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41923: connect: Connection refused (error 111)
I20260927 02:46:09.230248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 19868
I20260927 02:46:09.234894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 19506
2026-09-27T02:46:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2222 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T02:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:09Z Disabled control of system clock
I20260927 02:46:09.260010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36575
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:46:09.339941 20045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:09.340178 20045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:09.340198 20045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:09.341393 20045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:09.341441 20045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:09.341454 20045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:09.341464 20045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:09.341475 20045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:09.342841 20045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:09.343086 20045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:09.343376 20045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:09.348161 20051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:09.348147 20050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:09.348441 20045 server_base.cc:1034] running on GCE node
W20260927 02:46:09.348147 20053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:09.348752 20045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:09.349066 20045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:09.350224 20045 hybrid_clock.cc:648] HybridClock initialized: now 1790477169350189 us; error 46 us; skew 500 ppm
I20260927 02:46:09.351436 20045 webserver.cc:460] Webserver started at http://127.0.98.62:34699/ using document root <none> and password file <none>
I20260927 02:46:09.351667 20045 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:09.351716 20045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:09.351842 20045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:09.352955 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "81c62a1fbdee4d889e0e78c39718f748"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "f7ced3c3aa55d9b77b57ee7b85ca8be0"
server_key_iv: "a2b27e4065d3000b3bb14d07e0ddcc56"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.353293 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "81c62a1fbdee4d889e0e78c39718f748"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "f7ced3c3aa55d9b77b57ee7b85ca8be0"
server_key_iv: "a2b27e4065d3000b3bb14d07e0ddcc56"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.354614 20045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 02:46:09.355610 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.355872 20045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:09.356005 20045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "81c62a1fbdee4d889e0e78c39718f748"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "f7ced3c3aa55d9b77b57ee7b85ca8be0"
server_key_iv: "a2b27e4065d3000b3bb14d07e0ddcc56"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.356076 20045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:09.365355 20045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:09.365639 20045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:09.365741 20045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:09.369396 20045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36575
I20260927 02:46:09.369441 20111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36575 every 8 connection(s)
I20260927 02:46:09.369762 20045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:09.370721 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:09.373030 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748: Bootstrap starting.
I20260927 02:46:09.373588 20112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:09.373839 20112 log.cc:824] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:09.374313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 20045
I20260927 02:46:09.374402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:09.374532 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748: No bootstrap required, opened a new log
I20260927 02:46:09.374593 392 external_mini_cluster.cc:1428] Setting key dde4f9e9807ff39d517dc451afe0a1ca
I20260927 02:46:09.376529 20112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c62a1fbdee4d889e0e78c39718f748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } }
I20260927 02:46:09.376713 20112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:09.376748 20112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c62a1fbdee4d889e0e78c39718f748, State: Initialized, Role: FOLLOWER
I20260927 02:46:09.376897 20112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c62a1fbdee4d889e0e78c39718f748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } }
I20260927 02:46:09.376978 20112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:09.377010 20112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:09.377058 20112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:09.377897 20112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c62a1fbdee4d889e0e78c39718f748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } }
I20260927 02:46:09.377992 20112 leader_election.cc:302] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81c62a1fbdee4d889e0e78c39718f748; no voters:
I20260927 02:46:09.378157 20112 leader_election.cc:288] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:09.378206 20117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:09.378335 20117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [term 1 LEADER]: Becoming Leader. State: Replica: 81c62a1fbdee4d889e0e78c39718f748, State: Running, Role: LEADER
I20260927 02:46:09.378401 20112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:09.378418 20117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c62a1fbdee4d889e0e78c39718f748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } }
I20260927 02:46:09.378749 20117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81c62a1fbdee4d889e0e78c39718f748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c62a1fbdee4d889e0e78c39718f748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } } }
I20260927 02:46:09.378820 20117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:09.378942 20117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81c62a1fbdee4d889e0e78c39718f748. Latest consensus state: current_term: 1 leader_uuid: "81c62a1fbdee4d889e0e78c39718f748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c62a1fbdee4d889e0e78c39718f748" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } } }
I20260927 02:46:09.378995 20117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748 [sys.catalog]: This master's current role is: LEADER
W20260927 02:46:09.380216 20133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:09.380267 20133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:09.380471 20122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:09.380630 20122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:09.380967 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.375170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.0.98.62:36575)
0927 02:46:09.375395 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.375404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.375435 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:46:09.376484 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.376494 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.376574 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.376723 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:09.376962 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.377967 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.378887 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.379381 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.379510 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.380216 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:09.380220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:09.380220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:09.380438 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:09.380479 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:09.380493 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:09.380556 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.380664 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.380755 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260927 02:46:09.382136 20122 catalog_manager.cc:1347] Generated new cluster ID: d6c625b1792243e7afbdaf335e77e6db
I20260927 02:46:09.382185 20122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:09.391216 20122 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:09.391947 20122 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:09.401530 20122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748: Generated new TSK 0
I20260927 02:46:09.401767 20122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:09.405987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:46:09.487165 20136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:09.487377 20136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:09.487413 20136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:09.488592 20136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:09.488634 20136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:09.488667 20136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:46:09.490087 20136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:09.490345 20136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:09.490648 20136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:09.495514 20144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:09.495603 20142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:09.495514 20141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:09.496264 20136 server_base.cc:1034] running on GCE node
I20260927 02:46:09.496510 20136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:09.496755 20136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:09.497931 20136 hybrid_clock.cc:648] HybridClock initialized: now 1790477169497916 us; error 30 us; skew 500 ppm
I20260927 02:46:09.499225 20136 webserver.cc:460] Webserver started at http://127.0.98.1:44173/ using document root <none> and password file <none>
I20260927 02:46:09.499451 20136 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:09.499504 20136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:09.499658 20136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:09.500834 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/data/instance:
uuid: "d2a807cf931a4aafb32f8d30fccc7bed"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "648c19ea2f54ecfbfd579380faae7345"
server_key_iv: "287f1c8d00b4289b1bce44c2d0dce71d"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.501191 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d2a807cf931a4aafb32f8d30fccc7bed"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "648c19ea2f54ecfbfd579380faae7345"
server_key_iv: "287f1c8d00b4289b1bce44c2d0dce71d"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.502421 20136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:09.503154 20150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.503306 20136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:46:09.503384 20136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/wal
uuid: "d2a807cf931a4aafb32f8d30fccc7bed"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "648c19ea2f54ecfbfd579380faae7345"
server_key_iv: "287f1c8d00b4289b1bce44c2d0dce71d"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.503453 20136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:09.512843 20136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:09.513139 20136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:09.513271 20136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:09.513486 20136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:09.513796 20136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:09.513828 20136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.513857 20136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:09.513883 20136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.519233 20136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46651
I20260927 02:46:09.519272 20263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46651 every 8 connection(s)
I20260927 02:46:09.519587 20136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:46:09.520659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 20136
I20260927 02:46:09.520757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:46:09.520946 392 external_mini_cluster.cc:1428] Setting key 4ea633c0057ec6d1d77db9aad084596f
I20260927 02:46:09.521781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:46:09.525379 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.520287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53063 (local address 127.0.98.62:36575)
0927 02:46:09.520344 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.520346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.520577 (+ 231us) server_negotiation.cc:407] Connection header received
0927 02:46:09.521196 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.521198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.521217 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.521268 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:09.521954 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.522447 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.523787 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.524047 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.524116 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.525042 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:09.525043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:09.525044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:09.525049 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:09.525066 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:09.525071 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:09.525132 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.525260 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.525291 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:46:09.525580 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.520258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36575 (local address 127.0.98.1:53063)
0927 02:46:09.520455 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.520490 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.521098 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.521350 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.521354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.521364 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:09.521842 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.521846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.522529 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.522530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.523699 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.523702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.524151 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.524152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.524280 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.524868 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:46:09.524869 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:09.524910 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:46:09.524941 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:09.525159 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:09.525167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:09.525170 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.525252 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.525337 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260927 02:46:09.526201 20264 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36575
I20260927 02:46:09.526307 20264 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:09.526520 20264 heartbeater.cc:511] Master 127.0.98.62:36575 requested a full tablet report, sending...
I20260927 02:46:09.527037 20075 ts_manager.cc:194] Registered new tserver with Master: d2a807cf931a4aafb32f8d30fccc7bed (127.0.98.1:46651)
I20260927 02:46:09.527642 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53063
W20260927 02:46:09.602368 20267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:09.602576 20267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:09.602604 20267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:09.603734 20267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:09.603775 20267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:09.603808 20267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:46:09.605163 20267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:09.605394 20267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:09.605670 20267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:09.610905 20275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:09.610916 20272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:09.611068 20273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:09.611198 20267 server_base.cc:1034] running on GCE node
I20260927 02:46:09.611392 20267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:09.611629 20267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:09.612776 20267 hybrid_clock.cc:648] HybridClock initialized: now 1790477169612759 us; error 31 us; skew 500 ppm
I20260927 02:46:09.614122 20267 webserver.cc:460] Webserver started at http://127.0.98.2:35303/ using document root <none> and password file <none>
I20260927 02:46:09.614355 20267 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:09.614404 20267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:09.614537 20267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:09.615598 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5d2687b820b4c738d2455825575fac3"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "cf0a84bc7dbbbcdf2afd6848866baeb2"
server_key_iv: "89c2ce9dbf8ceac6dffaa5adb1e124bc"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.615921 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5d2687b820b4c738d2455825575fac3"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "cf0a84bc7dbbbcdf2afd6848866baeb2"
server_key_iv: "89c2ce9dbf8ceac6dffaa5adb1e124bc"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.617293 20267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:46:09.618117 20281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.618312 20267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:09.618407 20267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/wal
uuid: "f5d2687b820b4c738d2455825575fac3"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "cf0a84bc7dbbbcdf2afd6848866baeb2"
server_key_iv: "89c2ce9dbf8ceac6dffaa5adb1e124bc"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.618476 20267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:09.642230 20267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:09.642561 20267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:09.642694 20267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:09.642913 20267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:09.643260 20267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:09.643296 20267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.643326 20267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:09.643343 20267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.648959 20267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42437
I20260927 02:46:09.649073 20394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42437 every 8 connection(s)
I20260927 02:46:09.649348 20267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:46:09.654531 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.650113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41903 (local address 127.0.98.62:36575)
0927 02:46:09.650190 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.650192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.650371 (+ 179us) server_negotiation.cc:407] Connection header received
0927 02:46:09.650910 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.650911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.650924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.650976 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:09.651643 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.652209 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.653239 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.653429 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.653492 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.654271 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:09.654272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:09.654273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:09.654278 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:09.654292 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:09.654297 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:09.654338 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.654411 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.654439 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:46:09.654749 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.650049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36575 (local address 127.0.98.2:41903)
0927 02:46:09.650251 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.650285 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.650806 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.651053 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.651057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.651065 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:09.651554 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.651559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.652303 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.652304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.653166 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.653168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.653498 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.653499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.653621 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.654143 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:46:09.654144 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:09.654172 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:46:09.654196 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:09.654344 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:09.654348 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:09.654352 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.654410 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.654522 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260927 02:46:09.655273 20395 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36575
I20260927 02:46:09.655354 20395 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:09.655532 20395 heartbeater.cc:511] Master 127.0.98.62:36575 requested a full tablet report, sending...
I20260927 02:46:09.655999 20075 ts_manager.cc:194] Registered new tserver with Master: f5d2687b820b4c738d2455825575fac3 (127.0.98.2:42437)
I20260927 02:46:09.656467 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41903
I20260927 02:46:09.657111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 20267
I20260927 02:46:09.657234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:46:09.657445 392 external_mini_cluster.cc:1428] Setting key e520ae96579196f500d74262ac418498
I20260927 02:46:09.658269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--builtin_ntp_servers=127.0.98.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:46:09.737098 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:09.737308 20398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:09.737327 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:09.738456 20398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:09.738497 20398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:09.738520 20398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:46:09.739830 20398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:09.740058 20398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:09.740342 20398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:09.745240 20406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:09.745406 20398 server_base.cc:1034] running on GCE node
W20260927 02:46:09.745249 20404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:09.745252 20403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:09.745707 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:09.745947 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:09.747139 20398 hybrid_clock.cc:648] HybridClock initialized: now 1790477169747126 us; error 27 us; skew 500 ppm
I20260927 02:46:09.748517 20398 webserver.cc:460] Webserver started at http://127.0.98.3:37941/ using document root <none> and password file <none>
I20260927 02:46:09.748742 20398 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:09.748790 20398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:09.748896 20398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:09.750149 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f9921d7ef2c43cd981a63552f6b1646"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "e9a303a2fe5cb3a3c08b833a096f5fee"
server_key_iv: "13366e738dd0102159954403e905a4b7"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.750527 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f9921d7ef2c43cd981a63552f6b1646"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "e9a303a2fe5cb3a3c08b833a096f5fee"
server_key_iv: "13366e738dd0102159954403e905a4b7"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.752681 20398 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:46:09.753614 20412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.753764 20398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:09.753854 20398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/wal
uuid: "9f9921d7ef2c43cd981a63552f6b1646"
format_stamp: "Formatted at 2026-09-27 02:46:09 on dist-test-slave-44kr"
server_key: "e9a303a2fe5cb3a3c08b833a096f5fee"
server_key_iv: "13366e738dd0102159954403e905a4b7"
server_key_version: "encryptionkey@0"
I20260927 02:46:09.753934 20398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:09.790890 20398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:09.791224 20398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:09.791350 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:09.791590 20398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:46:09.791930 20398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:46:09.791965 20398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.791992 20398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:46:09.792011 20398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:09.797618 20398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41627
I20260927 02:46:09.797664 20525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41627 every 8 connection(s)
I20260927 02:46:09.797998 20398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:46:09.803324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 20398
I20260927 02:46:09.803360 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.798735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55385 (local address 127.0.98.62:36575)
0927 02:46:09.798818 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.798820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.799000 (+ 180us) server_negotiation.cc:407] Connection header received
0927 02:46:09.799492 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.799494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.799503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.799554 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:09.800226 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.800783 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.801794 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.802042 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.802097 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.803016 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:09.803017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:09.803017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:09.803022 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:09.803035 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:09.803039 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:09.803083 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.803242 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.803275 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:46:09.803443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:46:09.803468 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.798702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36575 (local address 127.0.98.3:55385)
0927 02:46:09.798876 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.798910 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.799395 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.799621 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.799624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.799633 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:09.800142 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.800147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.800883 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.800884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.801707 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.801709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.802151 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.802153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.802274 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.802848 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:46:09.802849 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:09.802878 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:46:09.802902 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:09.803141 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:09.803144 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:09.803147 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.803200 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.803285 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260927 02:46:09.803668 392 external_mini_cluster.cc:1428] Setting key c3892988d4769989eaa1a910234575c4
I20260927 02:46:09.803965 20526 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36575
I20260927 02:46:09.804064 20526 heartbeater.cc:464] Registering TS with master...
I20260927 02:46:09.804255 20526 heartbeater.cc:511] Master 127.0.98.62:36575 requested a full tablet report, sending...
I20260927 02:46:09.804726 20075 ts_manager.cc:194] Registered new tserver with Master: 9f9921d7ef2c43cd981a63552f6b1646 (127.0.98.3:41627)
I20260927 02:46:09.805110 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55385
I20260927 02:46:09.815755 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:46:09.820147 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.817219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.0.98.62:36575)
0927 02:46:09.817313 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.817315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.817427 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:46:09.817485 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.817486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.817499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.817546 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:09.817711 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.818245 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.819010 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.819205 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.819259 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.819895 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:09.819896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:09.819897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:09.819901 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:09.819918 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:09.819922 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:09.819962 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.820023 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.820057 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:46:09.821806 20075 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:46:09.823354 20075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:46:09.829603 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.825826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.0.98.2:42437)
0927 02:46:09.826020 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.826025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.826067 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:46:09.826131 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.826132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.826145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.826210 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:09.826365 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.827500 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.828726 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.829212 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.829276 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.829369 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.829473 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.829513 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
I20260927 02:46:09.829592 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.825653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42437 (local address 127.0.0.1:48898)
0927 02:46:09.826018 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.826027 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.826072 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.826270 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.826271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.826272 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:09.826299 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.826300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.827602 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.827603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.828646 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.828648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.829292 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.829293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.829373 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.829386 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.829450 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.829514 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":53,"threads_started":1}
I20260927 02:46:09.829702 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.825460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41627 (local address 127.0.0.1:41832)
0927 02:46:09.825620 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.825665 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.825730 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.826348 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.826349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.826351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:09.826411 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.826413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.827304 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.827305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.828590 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.828593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.829502 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.829503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.829560 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.829564 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.829591 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.829630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260927 02:46:09.830054 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.825637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.0.98.3:41627)
0927 02:46:09.826184 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.826190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.826199 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:46:09.826238 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.826240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.826255 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.826469 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:09.826513 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.827211 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.828675 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.829392 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.829818 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.829910 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.829935 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.829970 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":71,"threads_started":1}
I20260927 02:46:09.830441 20329 tablet_service.cc:1467] Processing CreateTablet for tablet c76a9f8269da4c90963c2d77131c04c0 (DEFAULT_TABLE table=test-table [id=b707fc5572d74d549a04282bee2cd5ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:09.830775 20329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76a9f8269da4c90963c2d77131c04c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:09.830720 20460 tablet_service.cc:1467] Processing CreateTablet for tablet c76a9f8269da4c90963c2d77131c04c0 (DEFAULT_TABLE table=test-table [id=b707fc5572d74d549a04282bee2cd5ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:09.830960 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.825462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46651 (local address 127.0.0.1:32946)
0927 02:46:09.825804 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.825814 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.825883 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.826207 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.826210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.826213 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:09.826279 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.826281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.828445 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.828446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.829462 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.829464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.830727 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.830728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.830793 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.830797 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.830843 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.830881 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260927 02:46:09.831050 20460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76a9f8269da4c90963c2d77131c04c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:09.831173 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.825581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.98.1:46651)
0927 02:46:09.825791 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.825796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.825870 (+ 74us) server_negotiation.cc:407] Connection header received
0927 02:46:09.826013 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.826015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.826032 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.826127 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:09.826360 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.827518 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.829541 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.830548 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.830613 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.830692 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.831057 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.831100 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260927 02:46:09.833485 20540 tablet_bootstrap.cc:492] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: Bootstrap starting.
I20260927 02:46:09.833884 20541 tablet_bootstrap.cc:492] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: Bootstrap starting.
I20260927 02:46:09.834024 20198 tablet_service.cc:1467] Processing CreateTablet for tablet c76a9f8269da4c90963c2d77131c04c0 (DEFAULT_TABLE table=test-table [id=b707fc5572d74d549a04282bee2cd5ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:46:09.834215 20540 tablet_bootstrap.cc:654] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:09.834355 20198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76a9f8269da4c90963c2d77131c04c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:09.834494 20541 tablet_bootstrap.cc:654] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:09.834515 20540 log.cc:824] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:09.834810 20541 log.cc:824] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:09.835204 20540 tablet_bootstrap.cc:492] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: No bootstrap required, opened a new log
I20260927 02:46:09.835271 20540 ts_tablet_manager.cc:1397] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:09.835523 20541 tablet_bootstrap.cc:492] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: No bootstrap required, opened a new log
I20260927 02:46:09.835589 20541 ts_tablet_manager.cc:1397] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:09.837242 20540 raft_consensus.cc:348] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } }
I20260927 02:46:09.837373 20540 raft_consensus.cc:374] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:09.837411 20540 raft_consensus.cc:729] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9921d7ef2c43cd981a63552f6b1646, State: Initialized, Role: FOLLOWER
I20260927 02:46:09.837546 20540 consensus_queue.cc:260] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } }
I20260927 02:46:09.837695 20544 tablet_bootstrap.cc:492] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed: Bootstrap starting.
I20260927 02:46:09.837785 20540 ts_tablet_manager.cc:1428] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:46:09.837905 20526 heartbeater.cc:503] Master 127.0.98.62:36575 was elected leader, sending a full tablet report...
I20260927 02:46:09.838009 20541 raft_consensus.cc:348] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } }
I20260927 02:46:09.838160 20541 raft_consensus.cc:374] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:09.838198 20541 raft_consensus.cc:729] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d2687b820b4c738d2455825575fac3, State: Initialized, Role: FOLLOWER
I20260927 02:46:09.838299 20544 tablet_bootstrap.cc:654] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:09.838305 20541 consensus_queue.cc:260] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } }
I20260927 02:46:09.838550 20541 ts_tablet_manager.cc:1428] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 02:46:09.838573 20395 heartbeater.cc:503] Master 127.0.98.62:36575 was elected leader, sending a full tablet report...
I20260927 02:46:09.838603 20544 log.cc:824] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:09.839401 20544 tablet_bootstrap.cc:492] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed: No bootstrap required, opened a new log
I20260927 02:46:09.839476 20544 ts_tablet_manager.cc:1397] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:46:09.841238 20544 raft_consensus.cc:348] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } }
I20260927 02:46:09.841363 20544 raft_consensus.cc:374] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:09.841383 20544 raft_consensus.cc:729] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a807cf931a4aafb32f8d30fccc7bed, State: Initialized, Role: FOLLOWER
I20260927 02:46:09.841476 20544 consensus_queue.cc:260] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } }
I20260927 02:46:09.841688 20544 ts_tablet_manager.cc:1428] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:46:09.841766 20264 heartbeater.cc:503] Master 127.0.98.62:36575 was elected leader, sending a full tablet report...
W20260927 02:46:09.899909 20396 tablet.cc:2369] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:46:09.941202 20548 raft_consensus.cc:482] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:46:09.941360 20548 raft_consensus.cc:504] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } }
I20260927 02:46:09.941785 20548 leader_election.cc:288] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f9921d7ef2c43cd981a63552f6b1646 (127.0.98.3:41627), f5d2687b820b4c738d2455825575fac3 (127.0.98.2:42437)
I20260927 02:46:09.944737 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.942133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42309 (local address 127.0.98.3:41627)
0927 02:46:09.942188 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.942190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.942317 (+ 127us) server_negotiation.cc:407] Connection header received
0927 02:46:09.942374 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.942375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.942383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.942433 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:09.942570 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.943040 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.944014 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.944434 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.944499 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.944510 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.944649 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.944684 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:46:09.944783 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.941974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42437 (local address 127.0.98.1:39879)
0927 02:46:09.942140 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.942153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.942217 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.942357 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.942359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.942361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:09.942438 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.942441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.943125 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.943126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.943894 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.943896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.944555 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.944555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.944611 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.944613 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.944653 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.944686 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:46:09.944788 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.942065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41627 (local address 127.0.98.1:42309)
0927 02:46:09.942241 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.942251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:09.942314 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:09.942440 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:09.942441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:09.942442 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:46:09.942506 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.942507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.943121 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.943122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.943925 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:09.943928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.944510 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.944511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:09.944589 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.944593 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:46:09.944645 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:46:09.944686 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":33,"thread_start_us":94,"threads_started":1}
I20260927 02:46:09.945032 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.942060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39879 (local address 127.0.98.2:42437)
0927 02:46:09.942116 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.942118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.942214 (+ 96us) server_negotiation.cc:407] Connection header received
0927 02:46:09.942280 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.942281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.942291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.942346 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:46:09.942511 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.943041 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.943988 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.944459 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.944520 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.944529 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.944940 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.944975 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:46:09.945161 20480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76a9f8269da4c90963c2d77131c04c0" candidate_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9921d7ef2c43cd981a63552f6b1646" is_pre_election: true
I20260927 02:46:09.945269 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76a9f8269da4c90963c2d77131c04c0" candidate_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d2687b820b4c738d2455825575fac3" is_pre_election: true
I20260927 02:46:09.945327 20480 raft_consensus.cc:2454] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a807cf931a4aafb32f8d30fccc7bed in term 0.
I20260927 02:46:09.945390 20349 raft_consensus.cc:2454] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a807cf931a4aafb32f8d30fccc7bed in term 0.
I20260927 02:46:09.945601 20154 leader_election.cc:302] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2a807cf931a4aafb32f8d30fccc7bed, f5d2687b820b4c738d2455825575fac3; no voters:
I20260927 02:46:09.945720 20548 raft_consensus.cc:2781] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:46:09.945793 20548 raft_consensus.cc:482] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:46:09.945813 20548 raft_consensus.cc:3037] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:09.946542 20548 raft_consensus.cc:504] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } }
I20260927 02:46:09.946664 20548 leader_election.cc:288] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [CANDIDATE]: Term 1 election: Requested vote from peers 9f9921d7ef2c43cd981a63552f6b1646 (127.0.98.3:41627), f5d2687b820b4c738d2455825575fac3 (127.0.98.2:42437)
I20260927 02:46:09.946803 20480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76a9f8269da4c90963c2d77131c04c0" candidate_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9921d7ef2c43cd981a63552f6b1646"
I20260927 02:46:09.946825 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76a9f8269da4c90963c2d77131c04c0" candidate_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d2687b820b4c738d2455825575fac3"
I20260927 02:46:09.946877 20480 raft_consensus.cc:3037] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:09.946895 20349 raft_consensus.cc:3037] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:09.947584 20480 raft_consensus.cc:2454] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a807cf931a4aafb32f8d30fccc7bed in term 1.
I20260927 02:46:09.947614 20349 raft_consensus.cc:2454] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a807cf931a4aafb32f8d30fccc7bed in term 1.
I20260927 02:46:09.947745 20153 leader_election.cc:302] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9921d7ef2c43cd981a63552f6b1646, d2a807cf931a4aafb32f8d30fccc7bed; no voters:
I20260927 02:46:09.947860 20548 raft_consensus.cc:2781] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:09.948011 20548 raft_consensus.cc:686] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 1 LEADER]: Becoming Leader. State: Replica: d2a807cf931a4aafb32f8d30fccc7bed, State: Running, Role: LEADER
I20260927 02:46:09.948113 20548 consensus_queue.cc:237] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } }
I20260927 02:46:09.948911 20076 catalog_manager.cc:5495] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed reported cstate change: term changed from 0 to 1, leader changed from <none> to d2a807cf931a4aafb32f8d30fccc7bed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2a807cf931a4aafb32f8d30fccc7bed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20260927 02:46:09.977901 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.975555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.0.98.1:46651)
0927 02:46:09.975607 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:09.975609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:09.975618 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:46:09.975699 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:09.975700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:09.975720 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:09.975773 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:09.975886 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.976417 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.977179 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:09.977372 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:09.977424 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:09.977563 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:09.977661 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:09.977719 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:09.977783 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:46:09.977802 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:46:09.980163 20480 raft_consensus.cc:1261] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [term 1 FOLLOWER]: Refusing update from remote peer d2a807cf931a4aafb32f8d30fccc7bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:09.980222 20349 raft_consensus.cc:1261] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [term 1 FOLLOWER]: Refusing update from remote peer d2a807cf931a4aafb32f8d30fccc7bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:09.980587 20550 consensus_queue.cc:1035] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d2687b820b4c738d2455825575fac3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:09.980710 20548 consensus_queue.cc:1035] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9921d7ef2c43cd981a63552f6b1646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:09.983178 20554 mvcc.cc:205] Tried to move back new op lower bound from 7333794488235307008 to 7333794488108068864. Current Snapshot: MvccSnapshot[applied={T|T < 7333794488235307008}]
I20260927 02:46:09.985255 20075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:09.985349 20075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:46:09.986055 20075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81c62a1fbdee4d889e0e78c39718f748: Sending DeleteTablet for 3 replicas of tablet c76a9f8269da4c90963c2d77131c04c0
I20260927 02:46:09.986395 20329 tablet_service.cc:1514] Processing DeleteTablet for tablet c76a9f8269da4c90963c2d77131c04c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:09 UTC) from {username='slave'} at 127.0.0.1:48898
I20260927 02:46:09.986416 20460 tablet_service.cc:1514] Processing DeleteTablet for tablet c76a9f8269da4c90963c2d77131c04c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:09 UTC) from {username='slave'} at 127.0.0.1:41832
I20260927 02:46:09.986389 20198 tablet_service.cc:1514] Processing DeleteTablet for tablet c76a9f8269da4c90963c2d77131c04c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:46:09 UTC) from {username='slave'} at 127.0.0.1:32946
I20260927 02:46:09.986625 20565 tablet_replica.cc:330] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed: stopping tablet replica
I20260927 02:46:09.986723 20565 raft_consensus.cc:2229] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:46:09.986909 20565 raft_consensus.cc:2258] T c76a9f8269da4c90963c2d77131c04c0 P d2a807cf931a4aafb32f8d30fccc7bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:09.986938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 20136
I20260927 02:46:09.987478 20564 tablet_replica.cc:330] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: stopping tablet replica
I20260927 02:46:09.987574 20564 raft_consensus.cc:2229] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:09.987663 20564 raft_consensus.cc:2258] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:09.988075 20564 ts_tablet_manager.cc:1905] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:09.988307 20563 tablet_replica.cc:330] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: stopping tablet replica
I20260927 02:46:09.988440 20563 raft_consensus.cc:2229] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:46:09.988556 20563 raft_consensus.cc:2258] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:46:09.989112 20563 ts_tablet_manager.cc:1905] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:46:09.989746 20564 ts_tablet_manager.cc:1918] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:09.989810 20564 log.cc:1196] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-2/wal/wals/c76a9f8269da4c90963c2d77131c04c0
I20260927 02:46:09.990046 20564 ts_tablet_manager.cc:1939] T c76a9f8269da4c90963c2d77131c04c0 P 9f9921d7ef2c43cd981a63552f6b1646: Deleting consensus metadata
I20260927 02:46:09.990581 20062 catalog_manager.cc:4841] TS 9f9921d7ef2c43cd981a63552f6b1646 (127.0.98.3:41627): tablet c76a9f8269da4c90963c2d77131c04c0 (table test-table [id=b707fc5572d74d549a04282bee2cd5ea]) successfully deleted
I20260927 02:46:09.990628 20563 ts_tablet_manager.cc:1918] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:46:09.990685 20563 log.cc:1196] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: Deleting WAL directory at /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476929252373-392-0/minicluster-data/ts-1/wal/wals/c76a9f8269da4c90963c2d77131c04c0
I20260927 02:46:09.990949 20563 ts_tablet_manager.cc:1939] T c76a9f8269da4c90963c2d77131c04c0 P f5d2687b820b4c738d2455825575fac3: Deleting consensus metadata
I20260927 02:46:09.991465 20063 catalog_manager.cc:4841] TS f5d2687b820b4c738d2455825575fac3 (127.0.98.2:42437): tablet c76a9f8269da4c90963c2d77131c04c0 (table test-table [id=b707fc5572d74d549a04282bee2cd5ea]) successfully deleted
W20260927 02:46:09.992322 20063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46651 (error 108)
I20260927 02:46:09.992806 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:09.992693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46651 (local address 127.0.0.1:32962)
0927 02:46:09.992743 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:09.992779 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 02:46:09.992913 20063 catalog_manager.cc:4584] TS d2a807cf931a4aafb32f8d30fccc7bed (127.0.98.1:46651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c76a9f8269da4c90963c2d77131c04c0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46651: connect: Connection refused (error 111)
I20260927 02:46:09.992973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 20267
I20260927 02:46:09.998025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 20398
I20260927 02:46:10.002599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 20045
2026-09-27T02:46:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (767 ms)
[----------] 6 tests from EncryptionPolicyTest (8390 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:10 dist-test-slave-44kr krb5kdc[20569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:10 dist-test-slave-44kr krb5kdc[20569](info): set up 2 sockets
Sep 27 02:46:10 dist-test-slave-44kr krb5kdc[20569](info): commencing operation
krb5kdc: starting...
W20260927 02:46:10.952097 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:11 dist-test-slave-44kr krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:11.075567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:11.155202 20582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:11.155411 20582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:11.155432 20582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:11.156634 20582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:11.156682 20582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:11.156695 20582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:11.156706 20582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:11.156718 20582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:11.158282 20582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:11.158525 20582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:11.158763 20582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:11.163664 20588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:11.163691 20590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:11.163671 20587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:11.163874 20582 server_base.cc:1034] running on GCE node
I20260927 02:46:11.164149 20582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:11.164505 20582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:11.165678 20582 hybrid_clock.cc:648] HybridClock initialized: now 1790477171165659 us; error 37 us; skew 500 ppm
Sep 27 02:46:11 dist-test-slave-44kr krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:11.168284 20582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:11.168728 20582 webserver.cc:460] Webserver started at http://127.0.0.1:37967/ using document root <none> and password file <none>
I20260927 02:46:11.168927 20582 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:11.168970 20582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:11.169057 20582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:11.170054 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "8d8e790b4e004c2c8cf23100e950daad"
format_stamp: "Formatted at 2026-09-27 02:46:11 on dist-test-slave-44kr"
server_key: "004cac18697811cbffa9769cdb130714"
server_key_iv: "5dbd5f1b8e46fc2122afe31a4bf4e77c"
server_key_version: "encryptionkey@0"
I20260927 02:46:11.170370 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d8e790b4e004c2c8cf23100e950daad"
format_stamp: "Formatted at 2026-09-27 02:46:11 on dist-test-slave-44kr"
server_key: "004cac18697811cbffa9769cdb130714"
server_key_iv: "5dbd5f1b8e46fc2122afe31a4bf4e77c"
server_key_version: "encryptionkey@0"
I20260927 02:46:11.171567 20582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:46:11.172263 20597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:11.172431 20582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:46:11.172508 20582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "8d8e790b4e004c2c8cf23100e950daad"
format_stamp: "Formatted at 2026-09-27 02:46:11 on dist-test-slave-44kr"
server_key: "004cac18697811cbffa9769cdb130714"
server_key_iv: "5dbd5f1b8e46fc2122afe31a4bf4e77c"
server_key_version: "encryptionkey@0"
I20260927 02:46:11.172559 20582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:11.187446 20582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:11.189540 20582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:11.189661 20582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:11.193447 20582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43755
I20260927 02:46:11.193480 20649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43755 every 8 connection(s)
I20260927 02:46:11.193805 20582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:11.194653 20650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:11.196980 20650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad: Bootstrap starting.
I20260927 02:46:11.197557 20650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:11.197809 20650 log.cc:824] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:11.198547 20650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad: No bootstrap required, opened a new log
I20260927 02:46:11.200182 20650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8e790b4e004c2c8cf23100e950daad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } }
I20260927 02:46:11.200325 20650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:11.200357 20650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8e790b4e004c2c8cf23100e950daad, State: Initialized, Role: FOLLOWER
I20260927 02:46:11.200526 20650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8e790b4e004c2c8cf23100e950daad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } }
I20260927 02:46:11.200609 20650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:11.200641 20650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:11.200690 20650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:11.200716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 20582
I20260927 02:46:11.200822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:11.200997 392 external_mini_cluster.cc:1428] Setting key 2a66863243523be1d5835cb6f1392d3e
I20260927 02:46:11.201298 20650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8e790b4e004c2c8cf23100e950daad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } }
I20260927 02:46:11.201397 20650 leader_election.cc:302] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d8e790b4e004c2c8cf23100e950daad; no voters:
I20260927 02:46:11.201615 20650 leader_election.cc:288] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:11.201663 20654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:11.201833 20654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [term 1 LEADER]: Becoming Leader. State: Replica: 8d8e790b4e004c2c8cf23100e950daad, State: Running, Role: LEADER
I20260927 02:46:11.201908 20650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:11.201965 20654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8e790b4e004c2c8cf23100e950daad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } }
I20260927 02:46:11.202265 20654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8e790b4e004c2c8cf23100e950daad. Latest consensus state: current_term: 1 leader_uuid: "8d8e790b4e004c2c8cf23100e950daad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8e790b4e004c2c8cf23100e950daad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } } }
I20260927 02:46:11.202339 20654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:11.202622 20656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d8e790b4e004c2c8cf23100e950daad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8e790b4e004c2c8cf23100e950daad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } } }
I20260927 02:46:11.202701 20656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad [sys.catalog]: This master's current role is: LEADER
W20260927 02:46:11.203543 20671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:11.203598 20671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:11.203649 20662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:11.203819 20662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:11.205294 20662 catalog_manager.cc:1347] Generated new cluster ID: b1f89cd4751c4da09324ab9d2e3a7b69
I20260927 02:46:11.205345 20662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:46:11 dist-test-slave-44kr krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:46:11.211699 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:11.201518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36963 (local address 127.0.0.1:43755)
0927 02:46:11.201693 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:11.201703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:11.201870 (+ 167us) server_negotiation.cc:407] Connection header received
0927 02:46:11.202016 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:11.202034 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:11.202092 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:11.202230 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:11.203520 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:11.204764 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:11.205657 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:11.206075 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:11.206174 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:11.208307 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:11.208311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:11.208311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:11.208323 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:11.210282 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:11.210634 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:11.210637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:11.210637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:11.210734 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:11.210930 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:11.210931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:11.210931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:11.211174 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:11.211318 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:11.211439 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 02:46:11.211535 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260927 02:46:11.212600 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:11.222337 20662 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:11.222862 20662 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:11.227771 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:11.216854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49547 (local address 127.0.0.1:43755)
0927 02:46:11.216897 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:11.216899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:11.220687 (+ 3788us) server_negotiation.cc:407] Connection header received
0927 02:46:11.220722 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:11.220723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:11.220735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:11.220793 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:11.221524 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:11.222115 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:11.223132 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:11.223445 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:11.223512 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:11.224968 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:11.224970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:11.224970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:11.224977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:11.226484 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:11.226952 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:11.226954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:11.226954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:11.227014 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:11.227225 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:11.227226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:11.227226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:11.227341 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:11.227456 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:11.227577 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 02:46:11.227644 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:46:11.227814 20662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d8e790b4e004c2c8cf23100e950daad: Generated new TSK 0
I20260927 02:46:11.228076 20662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:11.232786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 20582
W20260927 02:46:11.236604 20567 connection.cc:535] client connection to 127.0.0.1:43755 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43755): Connection reset by peer (error 104)
2026-09-27T02:46:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:11 dist-test-slave-44kr krb5kdc[20681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:11 dist-test-slave-44kr krb5kdc[20681](info): set up 2 sockets
Sep 27 02:46:11 dist-test-slave-44kr krb5kdc[20681](info): commencing operation
krb5kdc: starting...
W20260927 02:46:12.192173 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:12 dist-test-slave-44kr krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:12.312467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:12.392584 20694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:12.392810 20694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:12.392834 20694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:12.393989 20694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:12.394027 20694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:12.394040 20694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:12.394052 20694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:12.394062 20694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:12.395551 20694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:12.395754 20694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:12.395941 20694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:12.400889 20700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:12.400975 20699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:12.401075 20702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:12.401165 20694 server_base.cc:1034] running on GCE node
I20260927 02:46:12.401439 20694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:12.401764 20694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:12.402921 20694 hybrid_clock.cc:648] HybridClock initialized: now 1790477172402874 us; error 65 us; skew 500 ppm
Sep 27 02:46:12 dist-test-slave-44kr krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:12.408694 20694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:12.409145 20694 webserver.cc:460] Webserver started at http://127.0.0.1:34779/ using document root <none> and password file <none>
I20260927 02:46:12.409358 20694 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:12.409405 20694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:12.409535 20694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:12.410594 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "2dbd9220f5984111b09c6e65cfdcd1b7"
format_stamp: "Formatted at 2026-09-27 02:46:12 on dist-test-slave-44kr"
server_key: "188b4744211ebfdde1ddbf1b46125b74"
server_key_iv: "8a845914a0139fc18877642a2437e405"
server_key_version: "encryptionkey@0"
I20260927 02:46:12.410919 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "2dbd9220f5984111b09c6e65cfdcd1b7"
format_stamp: "Formatted at 2026-09-27 02:46:12 on dist-test-slave-44kr"
server_key: "188b4744211ebfdde1ddbf1b46125b74"
server_key_iv: "8a845914a0139fc18877642a2437e405"
server_key_version: "encryptionkey@0"
I20260927 02:46:12.412200 20694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:12.412984 20709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:12.413139 20694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:12.413223 20694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "2dbd9220f5984111b09c6e65cfdcd1b7"
format_stamp: "Formatted at 2026-09-27 02:46:12 on dist-test-slave-44kr"
server_key: "188b4744211ebfdde1ddbf1b46125b74"
server_key_iv: "8a845914a0139fc18877642a2437e405"
server_key_version: "encryptionkey@0"
I20260927 02:46:12.413295 20694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:12.431577 20694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:12.433365 20694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:12.433516 20694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:12.437333 20694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35893
I20260927 02:46:12.437381 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35893 every 8 connection(s)
I20260927 02:46:12.437731 20694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:12.438596 20762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:12.440944 20762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7: Bootstrap starting.
I20260927 02:46:12.441500 20762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:12.441746 20762 log.cc:824] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:12.442507 20762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7: No bootstrap required, opened a new log
I20260927 02:46:12.444129 20762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbd9220f5984111b09c6e65cfdcd1b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260927 02:46:12.444271 20762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:12.444293 20762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dbd9220f5984111b09c6e65cfdcd1b7, State: Initialized, Role: FOLLOWER
I20260927 02:46:12.444453 20762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbd9220f5984111b09c6e65cfdcd1b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260927 02:46:12.444520 20762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:12.444541 20762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:12.444568 20762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:12.445089 20762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbd9220f5984111b09c6e65cfdcd1b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260927 02:46:12.445173 20762 leader_election.cc:302] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dbd9220f5984111b09c6e65cfdcd1b7; no voters:
I20260927 02:46:12.445343 20762 leader_election.cc:288] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:12.445403 20765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:12.445588 20762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:12.445591 20765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [term 1 LEADER]: Becoming Leader. State: Replica: 2dbd9220f5984111b09c6e65cfdcd1b7, State: Running, Role: LEADER
I20260927 02:46:12.445708 20765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbd9220f5984111b09c6e65cfdcd1b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260927 02:46:12.446228 20766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dbd9220f5984111b09c6e65cfdcd1b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbd9220f5984111b09c6e65cfdcd1b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } } }
I20260927 02:46:12.446316 20766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:12.446473 20767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dbd9220f5984111b09c6e65cfdcd1b7. Latest consensus state: current_term: 1 leader_uuid: "2dbd9220f5984111b09c6e65cfdcd1b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbd9220f5984111b09c6e65cfdcd1b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } } }
I20260927 02:46:12.446535 20767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:12.446640 20774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:12.447099 20774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:12.447417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 20694
I20260927 02:46:12.447516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:12.447683 392 external_mini_cluster.cc:1428] Setting key 32a16d6e0b3495f7cbf795316c38715e
I20260927 02:46:12.448675 20774 catalog_manager.cc:1347] Generated new cluster ID: bc5795d97c464c1780714c1b4c4abfdd
I20260927 02:46:12.448724 20774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:12.454120 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:12.448184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57359 (local address 127.0.0.1:35893)
0927 02:46:12.448342 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:12.448352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:12.448467 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:46:12.448615 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:12.448623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:12.448683 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:12.448816 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:12.449145 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:12.450611 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:12.451675 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:12.452331 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:12.452480 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:12.453163 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:12.453167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:12.453167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:12.453377 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:12.453417 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:12.453433 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:12.453522 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:12.453695 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 02:46:12.453726 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260927 02:46:12.455295 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:12.457484 20774 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:12.458151 20774 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:12.463435 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:12.456778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39023 (local address 127.0.0.1:35893)
0927 02:46:12.456810 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:12.456812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:12.460641 (+ 3829us) server_negotiation.cc:407] Connection header received
0927 02:46:12.460705 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:12.460706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:12.460718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:12.460761 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:12.460886 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:12.461440 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:12.462301 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:12.462510 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:12.462559 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:12.463197 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:12.463198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:12.463198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:12.463203 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:12.463218 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:12.463223 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:12.463264 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:12.463346 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:46:12.463358 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:46:12.469030 20774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dbd9220f5984111b09c6e65cfdcd1b7: Generated new TSK 0
I20260927 02:46:12.469234 20774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:12.470904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 20694
W20260927 02:46:12.474530 20679 connection.cc:535] client connection to 127.0.0.1:35893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35893): Connection reset by peer (error 104)
2026-09-27T02:46:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:12 dist-test-slave-44kr krb5kdc[20793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:12 dist-test-slave-44kr krb5kdc[20793](info): set up 2 sockets
Sep 27 02:46:12 dist-test-slave-44kr krb5kdc[20793](info): commencing operation
krb5kdc: starting...
W20260927 02:46:13.416297 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:13 dist-test-slave-44kr krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:13.535563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:13.614813 20806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:13.615034 20806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:13.615057 20806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:13.616194 20806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:13.616232 20806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:13.616245 20806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:13.616255 20806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:13.616266 20806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:13.617810 20806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:13.618022 20806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:13.618196 20806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:13.623004 20812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:13.622991 20811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:13.622991 20814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:13.623262 20806 server_base.cc:1034] running on GCE node
I20260927 02:46:13.623571 20806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:13.623857 20806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:13.624995 20806 hybrid_clock.cc:648] HybridClock initialized: now 1790477173624968 us; error 43 us; skew 500 ppm
Sep 27 02:46:13 dist-test-slave-44kr krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:13.627408 20806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:13.627844 20806 webserver.cc:460] Webserver started at http://127.0.0.1:35079/ using document root <none> and password file <none>
I20260927 02:46:13.628118 20806 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:13.628173 20806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:13.628302 20806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:13.629423 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "96d883db9ed1498984a7edd6a51756fc"
format_stamp: "Formatted at 2026-09-27 02:46:13 on dist-test-slave-44kr"
server_key: "c3005564732b663fed50eb6cb1c2db03"
server_key_iv: "17602146b0b94332d05bb7cb7c776136"
server_key_version: "encryptionkey@0"
I20260927 02:46:13.629777 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "96d883db9ed1498984a7edd6a51756fc"
format_stamp: "Formatted at 2026-09-27 02:46:13 on dist-test-slave-44kr"
server_key: "c3005564732b663fed50eb6cb1c2db03"
server_key_iv: "17602146b0b94332d05bb7cb7c776136"
server_key_version: "encryptionkey@0"
I20260927 02:46:13.631165 20806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:46:13.632007 20821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:13.632200 20806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:13.632288 20806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "96d883db9ed1498984a7edd6a51756fc"
format_stamp: "Formatted at 2026-09-27 02:46:13 on dist-test-slave-44kr"
server_key: "c3005564732b663fed50eb6cb1c2db03"
server_key_iv: "17602146b0b94332d05bb7cb7c776136"
server_key_version: "encryptionkey@0"
I20260927 02:46:13.632364 20806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:13.635936 20806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:13.638064 20806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:13.638201 20806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:13.642035 20806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33481
I20260927 02:46:13.642082 20873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33481 every 8 connection(s)
I20260927 02:46:13.642387 20806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:13.643352 20874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:13.645720 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc: Bootstrap starting.
I20260927 02:46:13.646266 20874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:13.646490 20874 log.cc:824] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:13.647190 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc: No bootstrap required, opened a new log
I20260927 02:46:13.648797 20874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d883db9ed1498984a7edd6a51756fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260927 02:46:13.648938 20874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:13.648959 20874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96d883db9ed1498984a7edd6a51756fc, State: Initialized, Role: FOLLOWER
I20260927 02:46:13.649076 20874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d883db9ed1498984a7edd6a51756fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260927 02:46:13.649149 20874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:13.649178 20874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:13.649209 20874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:13.649691 20874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d883db9ed1498984a7edd6a51756fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260927 02:46:13.649776 20874 leader_election.cc:302] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96d883db9ed1498984a7edd6a51756fc; no voters:
I20260927 02:46:13.649914 20874 leader_election.cc:288] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:13.649977 20877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:13.650141 20874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:13.650352 20877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [term 1 LEADER]: Becoming Leader. State: Replica: 96d883db9ed1498984a7edd6a51756fc, State: Running, Role: LEADER
I20260927 02:46:13.650363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 20806
I20260927 02:46:13.650465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:13.650509 20877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d883db9ed1498984a7edd6a51756fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260927 02:46:13.650640 392 external_mini_cluster.cc:1428] Setting key e92a7f4e59014c15c77ac1469be8f129
I20260927 02:46:13.650975 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96d883db9ed1498984a7edd6a51756fc. Latest consensus state: current_term: 1 leader_uuid: "96d883db9ed1498984a7edd6a51756fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d883db9ed1498984a7edd6a51756fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260927 02:46:13.651060 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:13.651458 20878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96d883db9ed1498984a7edd6a51756fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d883db9ed1498984a7edd6a51756fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260927 02:46:13.651635 20878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc [sys.catalog]: This master's current role is: LEADER
W20260927 02:46:13.652086 20893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:13.652143 20893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:13.652223 20890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:13.652489 20890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:13.653471 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:13.651635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35017 (local address 127.0.0.1:33481)
0927 02:46:13.651818 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:13.651829 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:13.652000 (+ 171us) server_negotiation.cc:407] Connection header received
0927 02:46:13.652132 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:13.652138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:13.652183 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:13.652334 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:13.653047 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:13.653050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:13.653050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:13.653054 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:13.653099 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:13.653114 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:13.653184 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:13.653244 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 02:46:13.653294 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260927 02:46:13.653914 20890 catalog_manager.cc:1347] Generated new cluster ID: de62ce29d8a9499ab0d4091927ceac00
I20260927 02:46:13.653960 20890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:13.654260 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:13.657451 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:13.655760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51555 (local address 127.0.0.1:33481)
0927 02:46:13.655803 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:13.655805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:13.656567 (+ 762us) server_negotiation.cc:407] Connection header received
0927 02:46:13.656615 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:13.656616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:13.656621 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:13.656681 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:13.657245 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:13.657247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:13.657247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:13.657251 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:13.657269 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:13.657274 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:13.657331 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:13.657367 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:46:13.657380 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:46:13.660005 20890 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:13.660172 20890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:13.664134 20890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96d883db9ed1498984a7edd6a51756fc: Generated new TSK 0
I20260927 02:46:13.664351 20890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:13.668149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 20806
2026-09-27T02:46:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:13 dist-test-slave-44kr krb5kdc[20905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:13 dist-test-slave-44kr krb5kdc[20905](info): set up 2 sockets
Sep 27 02:46:13 dist-test-slave-44kr krb5kdc[20905](info): commencing operation
krb5kdc: starting...
W20260927 02:46:14.636243 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:14 dist-test-slave-44kr krb5kdc[20905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:14.757650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:14.840412 20918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:14.840646 20918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:14.840669 20918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:14.841935 20918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:14.841998 20918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:14.842012 20918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:14.842024 20918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:14.842034 20918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:14.843638 20918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:14.843914 20918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:14.844161 20918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:14.849279 20926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:14.849290 20924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:14.849290 20923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:14.849651 20918 server_base.cc:1034] running on GCE node
I20260927 02:46:14.849937 20918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:14.850266 20918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:14.851440 20918 hybrid_clock.cc:648] HybridClock initialized: now 1790477174851423 us; error 31 us; skew 500 ppm
Sep 27 02:46:14 dist-test-slave-44kr krb5kdc[20905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:14.854247 20918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:14.854708 20918 webserver.cc:460] Webserver started at http://127.0.0.1:44317/ using document root <none> and password file <none>
I20260927 02:46:14.854911 20918 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:14.854952 20918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:14.855049 20918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:14.856129 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "3cafeef3a45e4edca252b14fa54962f8"
format_stamp: "Formatted at 2026-09-27 02:46:14 on dist-test-slave-44kr"
server_key: "1e8f048a5015d5cac55c6678ccfd7fb4"
server_key_iv: "5b5190386d9c79316994f6bd25ac8825"
server_key_version: "encryptionkey@0"
I20260927 02:46:14.856499 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "3cafeef3a45e4edca252b14fa54962f8"
format_stamp: "Formatted at 2026-09-27 02:46:14 on dist-test-slave-44kr"
server_key: "1e8f048a5015d5cac55c6678ccfd7fb4"
server_key_iv: "5b5190386d9c79316994f6bd25ac8825"
server_key_version: "encryptionkey@0"
I20260927 02:46:14.857736 20918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:14.858526 20933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:14.858700 20918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:14.858788 20918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "3cafeef3a45e4edca252b14fa54962f8"
format_stamp: "Formatted at 2026-09-27 02:46:14 on dist-test-slave-44kr"
server_key: "1e8f048a5015d5cac55c6678ccfd7fb4"
server_key_iv: "5b5190386d9c79316994f6bd25ac8825"
server_key_version: "encryptionkey@0"
I20260927 02:46:14.858847 20918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:14.868224 20918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:14.870759 20918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:14.870894 20918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:14.874807 20918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33333
I20260927 02:46:14.874845 20985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33333 every 8 connection(s)
I20260927 02:46:14.875192 20918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:14.876048 20986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:14.878423 20986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8: Bootstrap starting.
I20260927 02:46:14.878991 20986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:14.879247 20986 log.cc:824] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:14.879907 20986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8: No bootstrap required, opened a new log
I20260927 02:46:14.881573 20986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cafeef3a45e4edca252b14fa54962f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } }
I20260927 02:46:14.881719 20986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:14.881750 20986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cafeef3a45e4edca252b14fa54962f8, State: Initialized, Role: FOLLOWER
I20260927 02:46:14.881886 20986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cafeef3a45e4edca252b14fa54962f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } }
I20260927 02:46:14.881947 20986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:14.881976 20986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:14.882025 20986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:14.882579 20986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cafeef3a45e4edca252b14fa54962f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } }
I20260927 02:46:14.882690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 20918
I20260927 02:46:14.882691 20986 leader_election.cc:302] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cafeef3a45e4edca252b14fa54962f8; no voters:
I20260927 02:46:14.882797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:14.882851 20986 leader_election.cc:288] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:14.882964 392 external_mini_cluster.cc:1428] Setting key 34a52ea07a3fffe0ef764c52e6d7559e
I20260927 02:46:14.882962 20989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:14.883155 20989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [term 1 LEADER]: Becoming Leader. State: Replica: 3cafeef3a45e4edca252b14fa54962f8, State: Running, Role: LEADER
I20260927 02:46:14.883222 20986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:14.883265 20989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cafeef3a45e4edca252b14fa54962f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } }
I20260927 02:46:14.883625 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cafeef3a45e4edca252b14fa54962f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cafeef3a45e4edca252b14fa54962f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } } }
I20260927 02:46:14.883711 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:14.883891 20995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:14.883857 20991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cafeef3a45e4edca252b14fa54962f8. Latest consensus state: current_term: 1 leader_uuid: "3cafeef3a45e4edca252b14fa54962f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cafeef3a45e4edca252b14fa54962f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } } }
I20260927 02:46:14.884008 20991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:14.884593 20995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:14.886373 20995 catalog_manager.cc:1347] Generated new cluster ID: 5f7210222b61449e97630af27bb150ee
I20260927 02:46:14.886421 20995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:46:14 dist-test-slave-44kr krb5kdc[20905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:46:14.894786 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:14.883451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38887 (local address 127.0.0.1:33333)
0927 02:46:14.884003 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:14.884013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:14.884046 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:46:14.884166 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:14.884173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:14.884230 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:14.884376 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:14.885306 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:14.886574 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:14.887639 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:14.888243 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:14.888327 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:14.891071 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:14.891075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:14.891076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:14.891087 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:14.893031 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:14.893408 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:14.893412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:14.893412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:14.893537 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:14.893713 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:14.893714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:14.893715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:14.894039 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:14.894204 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:14.894274 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:46:14.894535 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":79,"threads_started":1}
I20260927 02:46:14.895947 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:14.898108 20995 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:14.898589 20995 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:14.904203 20995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cafeef3a45e4edca252b14fa54962f8: Generated new TSK 0
I20260927 02:46:14.904542 20995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:14.904775 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:14.897427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56363 (local address 127.0.0.1:33333)
0927 02:46:14.897469 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:14.897471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:14.897672 (+ 201us) server_negotiation.cc:407] Connection header received
0927 02:46:14.897749 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:14.897751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:14.897762 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:14.897821 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:14.898428 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:14.899199 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:14.900240 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:14.900583 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:14.900624 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:14.902094 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:14.902096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:14.902096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:14.902104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:14.903501 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:14.903892 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:14.903893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:14.903894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:14.903950 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:14.904136 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:14.904137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:14.904138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:14.904267 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:14.904336 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:14.904584 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:46:14.904683 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:46:14.907066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 20918
2026-09-27T02:46:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:14 dist-test-slave-44kr krb5kdc[21017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:14 dist-test-slave-44kr krb5kdc[21017](info): set up 2 sockets
Sep 27 02:46:14 dist-test-slave-44kr krb5kdc[21017](info): commencing operation
krb5kdc: starting...
W20260927 02:46:15.869413 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:15 dist-test-slave-44kr krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:15.987079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:16.066433 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:16.066660 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:16.066684 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:16.067812 21030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:16.067854 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:16.067866 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:16.067878 21030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:16.067888 21030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:16.069756 21030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:16.069975 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:16.070173 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:16.074878 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:16.074973 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:16.074973 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:16.075181 21030 server_base.cc:1034] running on GCE node
I20260927 02:46:16.075448 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:16.075716 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:16.076870 21030 hybrid_clock.cc:648] HybridClock initialized: now 1790477176076824 us; error 59 us; skew 500 ppm
Sep 27 02:46:16 dist-test-slave-44kr krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:16.079428 21030 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:16.079864 21030 webserver.cc:460] Webserver started at http://127.0.0.1:37561/ using document root <none> and password file <none>
I20260927 02:46:16.080073 21030 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:16.080129 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:16.080258 21030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:16.081411 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "5382008c1e28465680455a2748f309f9"
format_stamp: "Formatted at 2026-09-27 02:46:16 on dist-test-slave-44kr"
server_key: "08db0e4d06d2e8adc54001f6b6c3a28a"
server_key_iv: "7fd1285dc3b79a22558772d67b3501e7"
server_key_version: "encryptionkey@0"
I20260927 02:46:16.081744 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "5382008c1e28465680455a2748f309f9"
format_stamp: "Formatted at 2026-09-27 02:46:16 on dist-test-slave-44kr"
server_key: "08db0e4d06d2e8adc54001f6b6c3a28a"
server_key_iv: "7fd1285dc3b79a22558772d67b3501e7"
server_key_version: "encryptionkey@0"
I20260927 02:46:16.083037 21030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:46:16.083788 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:16.084014 21030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:16.084131 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "5382008c1e28465680455a2748f309f9"
format_stamp: "Formatted at 2026-09-27 02:46:16 on dist-test-slave-44kr"
server_key: "08db0e4d06d2e8adc54001f6b6c3a28a"
server_key_iv: "7fd1285dc3b79a22558772d67b3501e7"
server_key_version: "encryptionkey@0"
I20260927 02:46:16.084192 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:16.102046 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:16.104224 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:16.104370 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:16.108168 21030 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37173
I20260927 02:46:16.108207 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37173 every 8 connection(s)
I20260927 02:46:16.108554 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:16.109426 21098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:16.111630 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9: Bootstrap starting.
I20260927 02:46:16.111861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 21030
I20260927 02:46:16.112030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:16.112193 392 external_mini_cluster.cc:1428] Setting key 22f124672cf8c287ef6a2bdc9ce988a0
I20260927 02:46:16.112265 21098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:16.112612 21098 log.cc:824] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:16.113323 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9: No bootstrap required, opened a new log
I20260927 02:46:16.115371 21098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5382008c1e28465680455a2748f309f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } }
I20260927 02:46:16.115547 21098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:16.115588 21098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5382008c1e28465680455a2748f309f9, State: Initialized, Role: FOLLOWER
I20260927 02:46:16.115716 21098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5382008c1e28465680455a2748f309f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } }
I20260927 02:46:16.115785 21098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:16.115814 21098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:16.115859 21098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:16.116446 21098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5382008c1e28465680455a2748f309f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } }
I20260927 02:46:16.116560 21098 leader_election.cc:302] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5382008c1e28465680455a2748f309f9; no voters:
I20260927 02:46:16.116742 21098 leader_election.cc:288] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:16.116911 21104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:16.117002 21098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:16.117049 21104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [term 1 LEADER]: Becoming Leader. State: Replica: 5382008c1e28465680455a2748f309f9, State: Running, Role: LEADER
I20260927 02:46:16.117156 21104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5382008c1e28465680455a2748f309f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } }
I20260927 02:46:16.117486 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5382008c1e28465680455a2748f309f9. Latest consensus state: current_term: 1 leader_uuid: "5382008c1e28465680455a2748f309f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5382008c1e28465680455a2748f309f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } } }
I20260927 02:46:16.117653 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:16.117599 21103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5382008c1e28465680455a2748f309f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5382008c1e28465680455a2748f309f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } } }
I20260927 02:46:16.117853 21103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:16.117913 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:16.112791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55319 (local address 127.0.0.1:37173)
0927 02:46:16.112948 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:16.112958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:16.112995 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:46:16.113495 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:16.113502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:16.113554 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:16.113682 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:16.113924 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:16.114840 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:16.115932 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:16.116363 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:16.116490 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:16.117231 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:16.117235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:16.117236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:16.117406 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:16.117452 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:16.117468 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:16.117527 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:16.117643 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 02:46:16.117713 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260927 02:46:16.119052 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 02:46:16.119297 21119 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:16.119349 21119 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:16.119401 21107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:16.119606 21107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:16.121484 21107 catalog_manager.cc:1347] Generated new cluster ID: f034347d3ed44ec285c3dd81c466588f
I20260927 02:46:16.121541 21107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:16.123335 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:16.120634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51307 (local address 127.0.0.1:37173)
0927 02:46:16.120675 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:16.120677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:16.120798 (+ 121us) server_negotiation.cc:407] Connection header received
0927 02:46:16.120877 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:16.120878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:16.120889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:16.120951 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:16.121064 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:16.121534 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:16.122251 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:16.122454 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:16.122508 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:16.123118 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:16.123119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:16.123119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:16.123124 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:16.123139 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:16.123142 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:16.123188 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:16.123243 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 02:46:16.123265 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:46:16.146751 21107 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:16.147315 21107 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:16.155246 21107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5382008c1e28465680455a2748f309f9: Generated new TSK 0
I20260927 02:46:16.155467 21107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:16.160295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 21030
2026-09-27T02:46:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:16 dist-test-slave-44kr krb5kdc[21129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:16 dist-test-slave-44kr krb5kdc[21129](info): set up 2 sockets
Sep 27 02:46:16 dist-test-slave-44kr krb5kdc[21129](info): commencing operation
krb5kdc: starting...
W20260927 02:46:17.120291 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:17 dist-test-slave-44kr krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:17.239624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:17.319420 21142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:17.319645 21142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:17.319677 21142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:17.320886 21142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:17.320930 21142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:17.320957 21142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:17.320974 21142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:17.320989 21142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:17.322458 21142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:17.322703 21142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:17.322924 21142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:17.327554 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:17.327577 21150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:17.327579 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:17.327735 21142 server_base.cc:1034] running on GCE node
I20260927 02:46:17.328066 21142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:17.328549 21142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:17.329722 21142 hybrid_clock.cc:648] HybridClock initialized: now 1790477177329708 us; error 31 us; skew 500 ppm
Sep 27 02:46:17 dist-test-slave-44kr krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:17.332155 21142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:17.332628 21142 webserver.cc:460] Webserver started at http://127.0.0.1:41673/ using document root <none> and password file <none>
I20260927 02:46:17.332835 21142 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:17.332882 21142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:17.333001 21142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:17.334045 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "60d085fc49424b21a54f7ab6dda1ca02"
format_stamp: "Formatted at 2026-09-27 02:46:17 on dist-test-slave-44kr"
server_key: "397b7effa9b9e96142cbcf0fc8bc9c92"
server_key_iv: "fa32646235aef0964d42fcb876b4f5a6"
server_key_version: "encryptionkey@0"
I20260927 02:46:17.334362 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "60d085fc49424b21a54f7ab6dda1ca02"
format_stamp: "Formatted at 2026-09-27 02:46:17 on dist-test-slave-44kr"
server_key: "397b7effa9b9e96142cbcf0fc8bc9c92"
server_key_iv: "fa32646235aef0964d42fcb876b4f5a6"
server_key_version: "encryptionkey@0"
I20260927 02:46:17.335629 21142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:17.336505 21157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:17.336668 21142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:17.336756 21142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "60d085fc49424b21a54f7ab6dda1ca02"
format_stamp: "Formatted at 2026-09-27 02:46:17 on dist-test-slave-44kr"
server_key: "397b7effa9b9e96142cbcf0fc8bc9c92"
server_key_iv: "fa32646235aef0964d42fcb876b4f5a6"
server_key_version: "encryptionkey@0"
I20260927 02:46:17.336820 21142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:17.340075 21142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:17.342236 21142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:17.342363 21142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:17.345969 21142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39887
I20260927 02:46:17.346048 21209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39887 every 8 connection(s)
I20260927 02:46:17.346324 21142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:17.347306 21210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:17.349534 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02: Bootstrap starting.
I20260927 02:46:17.350052 21210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:17.350291 21210 log.cc:824] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:17.350971 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02: No bootstrap required, opened a new log
I20260927 02:46:17.352603 21210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d085fc49424b21a54f7ab6dda1ca02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } }
I20260927 02:46:17.352751 21210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:17.352772 21210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d085fc49424b21a54f7ab6dda1ca02, State: Initialized, Role: FOLLOWER
I20260927 02:46:17.352887 21210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d085fc49424b21a54f7ab6dda1ca02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } }
I20260927 02:46:17.352957 21210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:17.352978 21210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:17.353006 21210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:17.353534 21210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d085fc49424b21a54f7ab6dda1ca02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } }
I20260927 02:46:17.353619 21210 leader_election.cc:302] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60d085fc49424b21a54f7ab6dda1ca02; no voters:
I20260927 02:46:17.353747 21210 leader_election.cc:288] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:17.353826 21213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:17.353981 21210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:17.354064 21213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [term 1 LEADER]: Becoming Leader. State: Replica: 60d085fc49424b21a54f7ab6dda1ca02, State: Running, Role: LEADER
I20260927 02:46:17.354161 21213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d085fc49424b21a54f7ab6dda1ca02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } }
I20260927 02:46:17.354249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 21142
I20260927 02:46:17.354357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:17.354516 392 external_mini_cluster.cc:1428] Setting key 135154d58393c34b68e1e525e296b6b8
I20260927 02:46:17.354542 21214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60d085fc49424b21a54f7ab6dda1ca02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d085fc49424b21a54f7ab6dda1ca02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } } }
I20260927 02:46:17.354638 21214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:17.354676 21215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60d085fc49424b21a54f7ab6dda1ca02. Latest consensus state: current_term: 1 leader_uuid: "60d085fc49424b21a54f7ab6dda1ca02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d085fc49424b21a54f7ab6dda1ca02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } } }
I20260927 02:46:17.354776 21215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:17.355222 21217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:17.355742 21217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:17.357426 21217 catalog_manager.cc:1347] Generated new cluster ID: aae778cc82814a2cb36793048bbaf821
I20260927 02:46:17.357509 21217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:17.358004 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:17.355062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48395 (local address 127.0.0.1:39887)
0927 02:46:17.356509 (+ 1447us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:17.356518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:17.356553 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:46:17.356657 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:17.356663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:17.356685 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:17.357084 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:17.357576 (+ 492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:17.357579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:17.357580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:17.357583 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:17.357615 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:17.357628 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:17.357680 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:17.357751 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:46:17.357869 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":70,"threads_started":1}
I20260927 02:46:17.358788 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:17.365082 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:17.360069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42647 (local address 127.0.0.1:39887)
0927 02:46:17.360110 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:17.360112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:17.360219 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:46:17.360273 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:17.360274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:17.360279 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:17.360321 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:17.364864 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:17.364866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:17.364866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:17.364871 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:17.364889 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:17.364892 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:17.364942 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:17.365000 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 02:46:17.365014 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:46:17.393271 21217 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:17.393501 21217 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:17.403690 21217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60d085fc49424b21a54f7ab6dda1ca02: Generated new TSK 0
I20260927 02:46:17.403945 21217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:17.414878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 21142
2026-09-27T02:46:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:17 dist-test-slave-44kr krb5kdc[21241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:17 dist-test-slave-44kr krb5kdc[21241](info): set up 2 sockets
Sep 27 02:46:17 dist-test-slave-44kr krb5kdc[21241](info): commencing operation
krb5kdc: starting...
W20260927 02:46:18.376343 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:18 dist-test-slave-44kr krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:18.496625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:18.574476 21254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:18.574739 21254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:18.574776 21254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:18.575982 21254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:18.576025 21254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:18.576045 21254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:18.576061 21254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:18.576088 21254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:18.577555 21254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:18.577800 21254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:18.578001 21254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:18.582646 21260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:18.582677 21262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:18.582677 21259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:18.582870 21254 server_base.cc:1034] running on GCE node
I20260927 02:46:18.583120 21254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:18.583381 21254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:18.584532 21254 hybrid_clock.cc:648] HybridClock initialized: now 1790477178584521 us; error 27 us; skew 500 ppm
Sep 27 02:46:18 dist-test-slave-44kr krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:18.587028 21254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:18.587440 21254 webserver.cc:460] Webserver started at http://127.0.0.1:43053/ using document root <none> and password file <none>
I20260927 02:46:18.587649 21254 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:18.587694 21254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:18.587798 21254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:18.588873 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "c03b38ea8e904b009853d555bf82c2fd"
format_stamp: "Formatted at 2026-09-27 02:46:18 on dist-test-slave-44kr"
server_key: "0c4643046b7bf50d28577d8d49d6ec91"
server_key_iv: "093af5a3a581f84a101dafd4b5d3cf59"
server_key_version: "encryptionkey@0"
I20260927 02:46:18.589206 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "c03b38ea8e904b009853d555bf82c2fd"
format_stamp: "Formatted at 2026-09-27 02:46:18 on dist-test-slave-44kr"
server_key: "0c4643046b7bf50d28577d8d49d6ec91"
server_key_iv: "093af5a3a581f84a101dafd4b5d3cf59"
server_key_version: "encryptionkey@0"
I20260927 02:46:18.590502 21254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:46:18.591266 21269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:18.591459 21254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:18.591552 21254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "c03b38ea8e904b009853d555bf82c2fd"
format_stamp: "Formatted at 2026-09-27 02:46:18 on dist-test-slave-44kr"
server_key: "0c4643046b7bf50d28577d8d49d6ec91"
server_key_iv: "093af5a3a581f84a101dafd4b5d3cf59"
server_key_version: "encryptionkey@0"
I20260927 02:46:18.591614 21254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:18.610903 21254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:18.613075 21254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:18.613193 21254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:18.616784 21254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42223
I20260927 02:46:18.616842 21321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42223 every 8 connection(s)
I20260927 02:46:18.617152 21254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:18.617996 21322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:18.620179 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd: Bootstrap starting.
I20260927 02:46:18.620751 21322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:18.620978 21322 log.cc:824] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:18.621088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 21254
I20260927 02:46:18.621202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:18.621363 392 external_mini_cluster.cc:1428] Setting key 266c692e4151df27027d57a763fcc6bb
I20260927 02:46:18.621692 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd: No bootstrap required, opened a new log
I20260927 02:46:18.623689 21322 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b38ea8e904b009853d555bf82c2fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260927 02:46:18.623858 21322 raft_consensus.cc:374] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:18.623895 21322 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03b38ea8e904b009853d555bf82c2fd, State: Initialized, Role: FOLLOWER
I20260927 02:46:18.624020 21322 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b38ea8e904b009853d555bf82c2fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260927 02:46:18.624100 21322 raft_consensus.cc:388] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:18.624137 21322 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:18.624183 21322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:18.624904 21322 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b38ea8e904b009853d555bf82c2fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260927 02:46:18.625013 21322 leader_election.cc:302] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c03b38ea8e904b009853d555bf82c2fd; no voters:
I20260927 02:46:18.625159 21322 leader_election.cc:288] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:18.625205 21327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:18.625375 21322 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:18.625413 21327 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [term 1 LEADER]: Becoming Leader. State: Replica: c03b38ea8e904b009853d555bf82c2fd, State: Running, Role: LEADER
I20260927 02:46:18.625558 21327 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b38ea8e904b009853d555bf82c2fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260927 02:46:18.625885 21328 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c03b38ea8e904b009853d555bf82c2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b38ea8e904b009853d555bf82c2fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } } }
I20260927 02:46:18.625955 21328 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:18.626154 21329 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03b38ea8e904b009853d555bf82c2fd. Latest consensus state: current_term: 1 leader_uuid: "c03b38ea8e904b009853d555bf82c2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b38ea8e904b009853d555bf82c2fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } } }
I20260927 02:46:18.626428 21329 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:18.626606 21335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:18.627323 21335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:18.629329 21335 catalog_manager.cc:1347] Generated new cluster ID: f1e730aae03d49838880a8863061b061
I20260927 02:46:18.629379 21335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:46:18 dist-test-slave-44kr krb5kdc[21241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:46:18.633476 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:18.621910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:47293 (local address 127.0.0.1:42223)
0927 02:46:18.622105 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:18.622115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:18.622150 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:46:18.622925 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:18.622935 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:18.623012 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:18.623201 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:18.623770 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:18.625491 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:18.626607 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:18.627124 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:18.627229 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:18.629967 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:18.629971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:18.629971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:18.629982 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:18.631754 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:18.632251 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:18.632254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:18.632254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:18.632355 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:18.632608 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:18.632609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:18.632609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:18.632892 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:18.633040 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:18.633190 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:46:18.633282 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260927 02:46:18.634378 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:18.650482 21335 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:18.651189 21335 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:18.655961 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:18.648841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:39897 (local address 127.0.0.1:42223)
0927 02:46:18.648902 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:18.648904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:18.649043 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:46:18.649109 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:18.649111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:18.649122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:18.649179 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:18.649871 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:18.650501 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:18.651477 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:18.651787 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:18.651859 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:18.653260 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:18.653261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:18.653262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:18.653269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:18.654820 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:18.655211 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:18.655212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:18.655213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:18.655273 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:18.655462 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:18.655463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:18.655463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:18.655586 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:18.655686 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:18.655794 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 02:46:18.655857 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:46:18.661562 21335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c03b38ea8e904b009853d555bf82c2fd: Generated new TSK 0
I20260927 02:46:18.661752 21335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:18.665102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 21254
W20260927 02:46:18.668912 21239 connection.cc:535] client connection to 127.0.0.1:42223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42223): Connection reset by peer (error 104)
2026-09-27T02:46:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:18 dist-test-slave-44kr krb5kdc[21353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:18 dist-test-slave-44kr krb5kdc[21353](info): set up 2 sockets
Sep 27 02:46:18 dist-test-slave-44kr krb5kdc[21353](info): commencing operation
krb5kdc: starting...
W20260927 02:46:19.616173 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:19 dist-test-slave-44kr krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:19.731886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:19.812026 21366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:19.812268 21366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:19.812305 21366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:19.813575 21366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:19.813627 21366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:19.813650 21366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:19.813669 21366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:19.813687 21366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:19.815292 21366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:19.815578 21366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:19.815809 21366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:19.820663 21374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:19.820641 21372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:19.820912 21366 server_base.cc:1034] running on GCE node
W20260927 02:46:19.820663 21371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:19.821241 21366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:19.821538 21366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:19.822712 21366 hybrid_clock.cc:648] HybridClock initialized: now 1790477179822704 us; error 34 us; skew 500 ppm
Sep 27 02:46:19 dist-test-slave-44kr krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:19.825421 21366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:19.825947 21366 webserver.cc:460] Webserver started at http://127.0.0.1:39825/ using document root <none> and password file <none>
I20260927 02:46:19.826157 21366 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:19.826203 21366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:19.826333 21366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:19.827425 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "9212065469194ff6b5a007ff56da537a"
format_stamp: "Formatted at 2026-09-27 02:46:19 on dist-test-slave-44kr"
server_key: "6f159d854bd0c6c7165c56ce417ae87d"
server_key_iv: "e9a4d8cacf9aad52b978c2917a62fa50"
server_key_version: "encryptionkey@0"
I20260927 02:46:19.827792 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "9212065469194ff6b5a007ff56da537a"
format_stamp: "Formatted at 2026-09-27 02:46:19 on dist-test-slave-44kr"
server_key: "6f159d854bd0c6c7165c56ce417ae87d"
server_key_iv: "e9a4d8cacf9aad52b978c2917a62fa50"
server_key_version: "encryptionkey@0"
I20260927 02:46:19.829289 21366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:19.830096 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:19.830267 21366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:19.830348 21366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "9212065469194ff6b5a007ff56da537a"
format_stamp: "Formatted at 2026-09-27 02:46:19 on dist-test-slave-44kr"
server_key: "6f159d854bd0c6c7165c56ce417ae87d"
server_key_iv: "e9a4d8cacf9aad52b978c2917a62fa50"
server_key_version: "encryptionkey@0"
I20260927 02:46:19.830412 21366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:19.845806 21366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:19.847702 21366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:19.847819 21366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:19.851642 21366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36137
I20260927 02:46:19.851676 21433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36137 every 8 connection(s)
I20260927 02:46:19.851994 21366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:19.852845 21434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:19.855127 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a: Bootstrap starting.
I20260927 02:46:19.855646 21434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:19.855861 21434 log.cc:824] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:19.856328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 21366
I20260927 02:46:19.856478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:19.856673 392 external_mini_cluster.cc:1428] Setting key 453fb7af61faeced3c767ce46b50c257
I20260927 02:46:19.856685 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a: No bootstrap required, opened a new log
I20260927 02:46:19.859555 21434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9212065469194ff6b5a007ff56da537a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } }
I20260927 02:46:19.859738 21434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:19.859771 21434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9212065469194ff6b5a007ff56da537a, State: Initialized, Role: FOLLOWER
I20260927 02:46:19.859911 21434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9212065469194ff6b5a007ff56da537a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } }
I20260927 02:46:19.859998 21434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:19.860041 21434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:19.860092 21434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:19.860736 21434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9212065469194ff6b5a007ff56da537a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } }
I20260927 02:46:19.860850 21434 leader_election.cc:302] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9212065469194ff6b5a007ff56da537a; no voters:
I20260927 02:46:19.861021 21434 leader_election.cc:288] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:19.861059 21439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:19.861207 21439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [term 1 LEADER]: Becoming Leader. State: Replica: 9212065469194ff6b5a007ff56da537a, State: Running, Role: LEADER
I20260927 02:46:19.861275 21434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:19.861336 21439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9212065469194ff6b5a007ff56da537a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } }
I20260927 02:46:19.861707 21439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9212065469194ff6b5a007ff56da537a. Latest consensus state: current_term: 1 leader_uuid: "9212065469194ff6b5a007ff56da537a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9212065469194ff6b5a007ff56da537a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } } }
I20260927 02:46:19.861814 21439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:19.861860 21441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9212065469194ff6b5a007ff56da537a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9212065469194ff6b5a007ff56da537a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } } }
I20260927 02:46:19.861930 21441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:19.862093 21445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:19.862634 21445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:19.863199 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:19.857296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:41045 (local address 127.0.0.1:36137)
0927 02:46:19.857478 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:19.857489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:19.857587 (+ 98us) server_negotiation.cc:407] Connection header received
0927 02:46:19.858032 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:19.858042 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:19.858111 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:19.858280 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:19.858504 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:19.859881 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:19.860794 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:19.861417 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:19.861574 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:19.862378 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:19.862382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:19.862382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:19.862610 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:19.862652 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:19.862668 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:19.862760 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:19.862846 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 02:46:19.862876 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260927 02:46:19.864204 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:19.864557 21445 catalog_manager.cc:1347] Generated new cluster ID: f1bc8214ea64495b85566eb8dfc92cc2
I20260927 02:46:19.864615 21445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:19.868157 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:19.865640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:56331 (local address 127.0.0.1:36137)
0927 02:46:19.865673 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:19.865675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:19.865682 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:46:19.865740 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:19.865740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:19.865749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:19.865798 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:19.865929 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:19.866386 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:19.867077 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:19.867294 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:19.867348 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:19.867950 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:19.867952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:19.867952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:19.867956 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:19.867971 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:19.867975 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:19.868023 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:19.868076 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 02:46:19.868085 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:46:19.873018 21445 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:19.873698 21445 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:19.879277 21445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9212065469194ff6b5a007ff56da537a: Generated new TSK 0
I20260927 02:46:19.879462 21445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:19.882494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 21366
W20260927 02:46:19.886405 21351 connection.cc:535] client connection to 127.0.0.1:36137 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36137): Connection reset by peer (error 104)
2026-09-27T02:46:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:19 dist-test-slave-44kr krb5kdc[21465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:19 dist-test-slave-44kr krb5kdc[21465](info): set up 2 sockets
Sep 27 02:46:19 dist-test-slave-44kr krb5kdc[21465](info): commencing operation
krb5kdc: starting...
W20260927 02:46:20.836323 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:20 dist-test-slave-44kr krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:20.955551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:21.037329 21478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:21.037557 21478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:21.037580 21478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:21.038731 21478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:21.038779 21478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:21.038794 21478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:21.038807 21478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:21.038822 21478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:21.040306 21478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:21.040575 21478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:21.040800 21478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:21.045825 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:21.045825 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:21.045830 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:21.045869 21478 server_base.cc:1034] running on GCE node
I20260927 02:46:21.046378 21478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:21.046671 21478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:21.047823 21478 hybrid_clock.cc:648] HybridClock initialized: now 1790477181047806 us; error 36 us; skew 500 ppm
Sep 27 02:46:21 dist-test-slave-44kr krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:21.050396 21478 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:21.050846 21478 webserver.cc:460] Webserver started at http://127.0.0.1:43059/ using document root <none> and password file <none>
I20260927 02:46:21.051046 21478 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:21.051090 21478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:21.051182 21478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:21.052209 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "b90b9abf17fd4098b10e5d17d5bde9e1"
format_stamp: "Formatted at 2026-09-27 02:46:21 on dist-test-slave-44kr"
server_key: "c712218ac60dbc263343723daebffa20"
server_key_iv: "b4f4f0c7cbec1abfdd618a9148404199"
server_key_version: "encryptionkey@0"
I20260927 02:46:21.052578 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "b90b9abf17fd4098b10e5d17d5bde9e1"
format_stamp: "Formatted at 2026-09-27 02:46:21 on dist-test-slave-44kr"
server_key: "c712218ac60dbc263343723daebffa20"
server_key_iv: "b4f4f0c7cbec1abfdd618a9148404199"
server_key_version: "encryptionkey@0"
I20260927 02:46:21.053825 21478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:21.054689 21493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:21.054950 21478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:21.055027 21478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "b90b9abf17fd4098b10e5d17d5bde9e1"
format_stamp: "Formatted at 2026-09-27 02:46:21 on dist-test-slave-44kr"
server_key: "c712218ac60dbc263343723daebffa20"
server_key_iv: "b4f4f0c7cbec1abfdd618a9148404199"
server_key_version: "encryptionkey@0"
I20260927 02:46:21.055106 21478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:21.058396 21478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:21.061045 21478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:21.061172 21478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:21.065397 21478 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39373
I20260927 02:46:21.065438 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39373 every 8 connection(s)
I20260927 02:46:21.065769 21478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:21.066767 21546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:21.069136 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1: Bootstrap starting.
I20260927 02:46:21.069702 21546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:21.069953 21546 log.cc:824] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:21.070394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 21478
I20260927 02:46:21.070514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:21.070670 392 external_mini_cluster.cc:1428] Setting key ed380ba0ec27960c196958178495d00a
I20260927 02:46:21.070758 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1: No bootstrap required, opened a new log
I20260927 02:46:21.072659 21546 raft_consensus.cc:348] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90b9abf17fd4098b10e5d17d5bde9e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39373 } }
I20260927 02:46:21.072814 21546 raft_consensus.cc:374] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:21.072849 21546 raft_consensus.cc:729] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90b9abf17fd4098b10e5d17d5bde9e1, State: Initialized, Role: FOLLOWER
I20260927 02:46:21.073015 21546 consensus_queue.cc:260] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90b9abf17fd4098b10e5d17d5bde9e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39373 } }
I20260927 02:46:21.073132 21546 raft_consensus.cc:388] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:21.073180 21546 raft_consensus.cc:482] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:21.073231 21546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:21.073872 21546 raft_consensus.cc:504] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90b9abf17fd4098b10e5d17d5bde9e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39373 } }
I20260927 02:46:21.073969 21546 leader_election.cc:302] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90b9abf17fd4098b10e5d17d5bde9e1; no voters:
I20260927 02:46:21.074134 21546 leader_election.cc:288] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:21.074252 21551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:21.074379 21546 sys_catalog.cc:564] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:21.074430 21551 raft_consensus.cc:686] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [term 1 LEADER]: Becoming Leader. State: Replica: b90b9abf17fd4098b10e5d17d5bde9e1, State: Running, Role: LEADER
I20260927 02:46:21.074402 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:21.071195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:47033 (local address 127.0.0.1:39373)
0927 02:46:21.071386 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:21.071394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:21.071421 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:46:21.072647 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:21.072656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:21.072691 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:21.072837 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:21.073862 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:21.073865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:21.073865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:21.073869 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:21.073903 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:21.073914 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:21.074018 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:21.074107 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:46:21.074157 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260927 02:46:21.074548 21551 consensus_queue.cc:237] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90b9abf17fd4098b10e5d17d5bde9e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39373 } }
I20260927 02:46:21.075098 21552 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b90b9abf17fd4098b10e5d17d5bde9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90b9abf17fd4098b10e5d17d5bde9e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39373 } } }
I20260927 02:46:21.075181 21552 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:21.075598 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:21.075596 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b90b9abf17fd4098b10e5d17d5bde9e1. Latest consensus state: current_term: 1 leader_uuid: "b90b9abf17fd4098b10e5d17d5bde9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90b9abf17fd4098b10e5d17d5bde9e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39373 } } }
I20260927 02:46:21.075659 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:21.075760 21560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:21.076227 21560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:21.077935 21560 catalog_manager.cc:1347] Generated new cluster ID: 0aa8152d5138479a9587ceb35a1edf6a
I20260927 02:46:21.078004 21560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:21.078524 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:21.077425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:53697 (local address 127.0.0.1:39373)
0927 02:46:21.077465 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:21.077467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:21.077605 (+ 138us) server_negotiation.cc:407] Connection header received
0927 02:46:21.077660 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:21.077661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:21.077667 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:21.077712 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:21.078314 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:21.078315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:21.078315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:21.078319 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:21.078333 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:21.078337 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:21.078362 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:21.078454 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:46:21.078466 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:46:21.086053 21560 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:21.086218 21560 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:21.091498 21560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b90b9abf17fd4098b10e5d17d5bde9e1: Generated new TSK 0
I20260927 02:46:21.091698 21560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:21.095232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 21478
2026-09-27T02:46:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:21 dist-test-slave-44kr krb5kdc[21577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:21 dist-test-slave-44kr krb5kdc[21577](info): set up 2 sockets
Sep 27 02:46:21 dist-test-slave-44kr krb5kdc[21577](info): commencing operation
krb5kdc: starting...
W20260927 02:46:22.056368 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:22 dist-test-slave-44kr krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:22.175313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:22.261159 21590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:22.261372 21590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:22.261402 21590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:22.262575 21590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:22.262619 21590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:22.262636 21590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:22.262651 21590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:22.262665 21590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:22.264173 21590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42597
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:22.264462 21590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:22.264678 21590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:46:22.269665 21590 server_base.cc:1034] running on GCE node
W20260927 02:46:22.269644 21596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:22.269646 21598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:22.269651 21595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:22.270090 21590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:22.270364 21590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:22.271584 21590 hybrid_clock.cc:648] HybridClock initialized: now 1790477182271559 us; error 44 us; skew 500 ppm
Sep 27 02:46:22 dist-test-slave-44kr krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:22.274328 21590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:22.274791 21590 webserver.cc:460] Webserver started at http://127.0.0.1:42821/ using document root <none> and password file <none>
I20260927 02:46:22.275121 21590 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:22.275261 21590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:22.275457 21590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:22.276873 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "17f4e269deee4f95a3ae0f337cc4a9e4"
format_stamp: "Formatted at 2026-09-27 02:46:22 on dist-test-slave-44kr"
server_key: "e86c5cf9e67f0096a8b740ce8e745335"
server_key_iv: "f9335c62c9d87e3c3b393b0b834ce83b"
server_key_version: "encryptionkey@0"
I20260927 02:46:22.277310 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "17f4e269deee4f95a3ae0f337cc4a9e4"
format_stamp: "Formatted at 2026-09-27 02:46:22 on dist-test-slave-44kr"
server_key: "e86c5cf9e67f0096a8b740ce8e745335"
server_key_iv: "f9335c62c9d87e3c3b393b0b834ce83b"
server_key_version: "encryptionkey@0"
I20260927 02:46:22.278829 21590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:46:22.279618 21605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:22.279809 21590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:22.279901 21590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "17f4e269deee4f95a3ae0f337cc4a9e4"
format_stamp: "Formatted at 2026-09-27 02:46:22 on dist-test-slave-44kr"
server_key: "e86c5cf9e67f0096a8b740ce8e745335"
server_key_iv: "f9335c62c9d87e3c3b393b0b834ce83b"
server_key_version: "encryptionkey@0"
I20260927 02:46:22.279960 21590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:22.288676 21590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:22.290753 21590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:22.290866 21590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:22.294601 21590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42597
I20260927 02:46:22.294644 21657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42597 every 8 connection(s)
I20260927 02:46:22.294963 21590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:22.295761 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:22.298187 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4: Bootstrap starting.
I20260927 02:46:22.298744 21658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:22.298997 21658 log.cc:824] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:22.299679 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4: No bootstrap required, opened a new log
I20260927 02:46:22.299717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 21590
I20260927 02:46:22.299820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:22.299984 392 external_mini_cluster.cc:1428] Setting key c24676d3cc552abc829d6ae4a45e791f
I20260927 02:46:22.301529 21658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4e269deee4f95a3ae0f337cc4a9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42597 } }
I20260927 02:46:22.301666 21658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:22.301687 21658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f4e269deee4f95a3ae0f337cc4a9e4, State: Initialized, Role: FOLLOWER
I20260927 02:46:22.301817 21658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4e269deee4f95a3ae0f337cc4a9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42597 } }
I20260927 02:46:22.301900 21658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:22.301929 21658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:22.301976 21658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:22.302922 21658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4e269deee4f95a3ae0f337cc4a9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42597 } }
I20260927 02:46:22.303011 21658 leader_election.cc:302] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17f4e269deee4f95a3ae0f337cc4a9e4; no voters:
I20260927 02:46:22.303154 21658 leader_election.cc:288] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:22.303193 21663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:22.303346 21663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [term 1 LEADER]: Becoming Leader. State: Replica: 17f4e269deee4f95a3ae0f337cc4a9e4, State: Running, Role: LEADER
I20260927 02:46:22.303407 21658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:22.303427 21663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4e269deee4f95a3ae0f337cc4a9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42597 } }
I20260927 02:46:22.305015 21665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17f4e269deee4f95a3ae0f337cc4a9e4. Latest consensus state: current_term: 1 leader_uuid: "17f4e269deee4f95a3ae0f337cc4a9e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4e269deee4f95a3ae0f337cc4a9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42597 } } }
I20260927 02:46:22.305027 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17f4e269deee4f95a3ae0f337cc4a9e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4e269deee4f95a3ae0f337cc4a9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42597 } } }
I20260927 02:46:22.305769 21665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [sys.catalog]: This master's current role is: LEADER
W20260927 02:46:22.305243 21678 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 02:46:22.305780 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4 [sys.catalog]: This master's current role is: LEADER
W20260927 02:46:22.305850 21678 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:22.305990 21680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:22.306212 21680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:22.307720 21680 catalog_manager.cc:1347] Generated new cluster ID: 4d794b6e73ed448d917d3f7ba0e8c3d1
I20260927 02:46:22.307770 21680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:46:22 dist-test-slave-44kr krb5kdc[21577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:46:22.318225 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:22.300498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:58807 (local address 127.0.0.1:42597)
0927 02:46:22.300698 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:22.300708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:22.300734 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:46:22.301537 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:22.301545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:22.301616 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:22.301767 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:22.302531 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:22.303672 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:22.304791 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:22.305363 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:22.305445 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:22.314442 (+ 8997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:22.314447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:22.314448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:22.314463 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:22.316470 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:22.316948 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:22.316951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:22.316951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:22.317072 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:22.317285 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:22.317285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:22.317286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:22.317603 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:22.317748 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:22.317866 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 02:46:22.317982 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260927 02:46:22.319209 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:22.320657 21680 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:22.321372 21680 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:22.323685 21680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17f4e269deee4f95a3ae0f337cc4a9e4: Generated new TSK 0
I20260927 02:46:22.323902 21680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:22.326949 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:22.320782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:41769 (local address 127.0.0.1:42597)
0927 02:46:22.320821 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:22.320823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:22.320985 (+ 162us) server_negotiation.cc:407] Connection header received
0927 02:46:22.321038 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:22.321039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:22.321049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:22.321089 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:22.321573 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:22.322107 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:22.322795 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:22.323041 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:22.323096 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:22.324310 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:22.324311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:22.324311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:22.324318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:22.325708 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:22.326135 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:22.326136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:22.326136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:22.326199 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:22.326421 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:22.326422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:22.326422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:22.326537 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:22.326633 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:22.326781 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 02:46:22.326845 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:46:22.329003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 21590
W20260927 02:46:22.332825 21575 connection.cc:535] client connection to 127.0.0.1:42597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42597): Connection reset by peer (error 104)
2026-09-27T02:46:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:22 dist-test-slave-44kr krb5kdc[21689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:22 dist-test-slave-44kr krb5kdc[21689](info): set up 2 sockets
Sep 27 02:46:22 dist-test-slave-44kr krb5kdc[21689](info): commencing operation
krb5kdc: starting...
W20260927 02:46:23.292428 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:23 dist-test-slave-44kr krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:23.411819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:23.490377 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:23.490600 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:23.490633 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:23.491791 21702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:23.491837 21702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:23.491854 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:23.491870 21702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:23.491885 21702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:23.493419 21702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:23.493686 21702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:23.493911 21702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:23.498715 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:23.498718 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:23.498718 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:23.498924 21702 server_base.cc:1034] running on GCE node
I20260927 02:46:23.499265 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:23.499557 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:23.500710 21702 hybrid_clock.cc:648] HybridClock initialized: now 1790477183500697 us; error 34 us; skew 500 ppm
Sep 27 02:46:23 dist-test-slave-44kr krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:23.509341 21702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:23.509872 21702 webserver.cc:460] Webserver started at http://127.0.0.1:42561/ using document root <none> and password file <none>
I20260927 02:46:23.510106 21702 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:23.510159 21702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:23.510285 21702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:23.511406 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "391b760ababf47ea94f591fe459cac21"
format_stamp: "Formatted at 2026-09-27 02:46:23 on dist-test-slave-44kr"
server_key: "954ff1eeaa171da808b994b4902a4296"
server_key_iv: "208193d754dd16204bdf3d6996100879"
server_key_version: "encryptionkey@0"
I20260927 02:46:23.511745 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "391b760ababf47ea94f591fe459cac21"
format_stamp: "Formatted at 2026-09-27 02:46:23 on dist-test-slave-44kr"
server_key: "954ff1eeaa171da808b994b4902a4296"
server_key_iv: "208193d754dd16204bdf3d6996100879"
server_key_version: "encryptionkey@0"
I20260927 02:46:23.513238 21702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:46:23.514058 21717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:23.514334 21702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:23.514490 21702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "391b760ababf47ea94f591fe459cac21"
format_stamp: "Formatted at 2026-09-27 02:46:23 on dist-test-slave-44kr"
server_key: "954ff1eeaa171da808b994b4902a4296"
server_key_iv: "208193d754dd16204bdf3d6996100879"
server_key_version: "encryptionkey@0"
I20260927 02:46:23.514571 21702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:23.552366 21702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:23.554550 21702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:23.554711 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:23.558727 21702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38377
I20260927 02:46:23.558786 21769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38377 every 8 connection(s)
I20260927 02:46:23.559132 21702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:23.560123 21770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:23.562413 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21: Bootstrap starting.
I20260927 02:46:23.562959 21770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:23.563194 21770 log.cc:824] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:23.563851 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21: No bootstrap required, opened a new log
I20260927 02:46:23.565415 21770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391b760ababf47ea94f591fe459cac21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } }
I20260927 02:46:23.565594 21770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:23.565618 21770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391b760ababf47ea94f591fe459cac21, State: Initialized, Role: FOLLOWER
I20260927 02:46:23.565727 21770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391b760ababf47ea94f591fe459cac21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } }
I20260927 02:46:23.565776 21770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:23.565800 21770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:23.565829 21770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:23.566316 21770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391b760ababf47ea94f591fe459cac21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } }
I20260927 02:46:23.566397 21770 leader_election.cc:302] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391b760ababf47ea94f591fe459cac21; no voters:
I20260927 02:46:23.566530 21770 leader_election.cc:288] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:23.566581 21773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:23.566762 21770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:23.566809 21773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [term 1 LEADER]: Becoming Leader. State: Replica: 391b760ababf47ea94f591fe459cac21, State: Running, Role: LEADER
I20260927 02:46:23.566913 21773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391b760ababf47ea94f591fe459cac21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } }
I20260927 02:46:23.566947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 21702
I20260927 02:46:23.567075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:23.567281 392 external_mini_cluster.cc:1428] Setting key bf65dbc4803d37822293be9eba0068bc
I20260927 02:46:23.567276 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391b760ababf47ea94f591fe459cac21. Latest consensus state: current_term: 1 leader_uuid: "391b760ababf47ea94f591fe459cac21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391b760ababf47ea94f591fe459cac21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } } }
I20260927 02:46:23.567350 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:23.567409 21774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "391b760ababf47ea94f591fe459cac21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391b760ababf47ea94f591fe459cac21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } } }
I20260927 02:46:23.567457 21774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:23.567552 21782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:23.568084 21782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:23.569657 21782 catalog_manager.cc:1347] Generated new cluster ID: 41e8613785394e718140483ede65eb13
I20260927 02:46:23.569708 21782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:23.573834 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:23.568105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:44119 (local address 127.0.0.1:38377)
0927 02:46:23.568379 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:23.568412 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:23.568446 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:46:23.568756 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:23.568764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:23.568828 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:23.568972 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:23.569215 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:23.570566 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:23.571677 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:23.572309 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:23.572460 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:23.573127 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:23.573132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:23.573132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:23.573358 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:23.573398 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:23.573413 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:23.573514 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:23.573579 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:46:23.573608 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
I20260927 02:46:23.574810 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:23.580032 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:23.576333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:48499 (local address 127.0.0.1:38377)
0927 02:46:23.576369 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:23.576370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:23.576464 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:46:23.576528 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:23.576529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:23.576540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:23.576595 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:23.576778 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:23.577522 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:23.578589 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:23.578869 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:23.578922 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:23.579721 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:23.579723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:23.579723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:23.579727 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:23.579743 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:23.579747 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:23.579792 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:23.579937 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:46:23.579949 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:46:23.580360 21782 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:23.580889 21782 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:23.588953 21782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 391b760ababf47ea94f591fe459cac21: Generated new TSK 0
I20260927 02:46:23.589187 21782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:23.591069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 21702
W20260927 02:46:23.594708 21687 connection.cc:535] client connection to 127.0.0.1:38377 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38377): Connection reset by peer (error 104)
2026-09-27T02:46:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:23 dist-test-slave-44kr krb5kdc[21801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:23 dist-test-slave-44kr krb5kdc[21801](info): set up 2 sockets
Sep 27 02:46:23 dist-test-slave-44kr krb5kdc[21801](info): commencing operation
krb5kdc: starting...
W20260927 02:46:24.544358 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:24 dist-test-slave-44kr krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:46:24.660280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:24.741075 21814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:24.741317 21814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:24.741349 21814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:24.742589 21814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:24.742648 21814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:24.742666 21814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:24.742683 21814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:24.742697 21814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:24.744328 21814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:24.744623 21814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:24.744843 21814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:24.749778 21820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:24.749783 21819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:24.749783 21822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:24.749943 21814 server_base.cc:1034] running on GCE node
I20260927 02:46:24.750324 21814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:24.750640 21814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:24.751806 21814 hybrid_clock.cc:648] HybridClock initialized: now 1790477184751793 us; error 34 us; skew 500 ppm
Sep 27 02:46:24 dist-test-slave-44kr krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:24.754309 21814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:46:24.754765 21814 webserver.cc:460] Webserver started at http://127.0.0.1:39067/ using document root <none> and password file <none>
I20260927 02:46:24.754974 21814 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:24.755015 21814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:24.755110 21814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:24.756134 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "4b382a48364d4f5fa0fcdae87d14f7b8"
format_stamp: "Formatted at 2026-09-27 02:46:24 on dist-test-slave-44kr"
server_key: "171b77f06be87a4dc99515323b6947fb"
server_key_iv: "b352f0094b525c4e162c15ddb061038c"
server_key_version: "encryptionkey@0"
I20260927 02:46:24.756489 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b382a48364d4f5fa0fcdae87d14f7b8"
format_stamp: "Formatted at 2026-09-27 02:46:24 on dist-test-slave-44kr"
server_key: "171b77f06be87a4dc99515323b6947fb"
server_key_iv: "b352f0094b525c4e162c15ddb061038c"
server_key_version: "encryptionkey@0"
I20260927 02:46:24.757764 21814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:24.758507 21829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:24.758718 21814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:24.758814 21814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "4b382a48364d4f5fa0fcdae87d14f7b8"
format_stamp: "Formatted at 2026-09-27 02:46:24 on dist-test-slave-44kr"
server_key: "171b77f06be87a4dc99515323b6947fb"
server_key_iv: "b352f0094b525c4e162c15ddb061038c"
server_key_version: "encryptionkey@0"
I20260927 02:46:24.758942 21814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:24.762447 21814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:24.764777 21814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:24.764904 21814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:24.768671 21814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36329
I20260927 02:46:24.768704 21881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36329 every 8 connection(s)
I20260927 02:46:24.769037 21814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:24.770033 21882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:24.772264 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8: Bootstrap starting.
I20260927 02:46:24.772850 21882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:24.773105 21882 log.cc:824] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:24.773811 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8: No bootstrap required, opened a new log
I20260927 02:46:24.775024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 21814
I20260927 02:46:24.775122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:24.775287 392 external_mini_cluster.cc:1428] Setting key 3d315dda41c25067e3bf3f1811436dd1
I20260927 02:46:24.775502 21882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b382a48364d4f5fa0fcdae87d14f7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36329 } }
I20260927 02:46:24.775640 21882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:24.775672 21882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b382a48364d4f5fa0fcdae87d14f7b8, State: Initialized, Role: FOLLOWER
I20260927 02:46:24.776072 21882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b382a48364d4f5fa0fcdae87d14f7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36329 } }
I20260927 02:46:24.776156 21882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:24.776189 21882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:24.776237 21882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:24.777331 21882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b382a48364d4f5fa0fcdae87d14f7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36329 } }
I20260927 02:46:24.777452 21882 leader_election.cc:302] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b382a48364d4f5fa0fcdae87d14f7b8; no voters:
I20260927 02:46:24.777616 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:24.775824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:58093 (local address 127.0.0.1:36329)
0927 02:46:24.776020 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:24.776028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:24.776067 (+ 39us) server_negotiation.cc:407] Connection header received
0927 02:46:24.776181 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:24.776189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:24.776222 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:24.776429 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:24.777094 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:24.777098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:24.777098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:24.777102 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:24.777145 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:24.777156 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:24.777218 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:24.777303 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:46:24.777341 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260927 02:46:24.777638 21882 leader_election.cc:288] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:24.777667 21887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:24.777768 21887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [term 1 LEADER]: Becoming Leader. State: Replica: 4b382a48364d4f5fa0fcdae87d14f7b8, State: Running, Role: LEADER
I20260927 02:46:24.777848 21887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b382a48364d4f5fa0fcdae87d14f7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36329 } }
I20260927 02:46:24.778070 21882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:24.778162 21887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b382a48364d4f5fa0fcdae87d14f7b8. Latest consensus state: current_term: 1 leader_uuid: "4b382a48364d4f5fa0fcdae87d14f7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b382a48364d4f5fa0fcdae87d14f7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36329 } } }
I20260927 02:46:24.778239 21887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:24.778391 21887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b382a48364d4f5fa0fcdae87d14f7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b382a48364d4f5fa0fcdae87d14f7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36329 } } }
I20260927 02:46:24.778446 21887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:24.778548 21891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:24.778558 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:24.779037 21891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:24.780824 21891 catalog_manager.cc:1347] Generated new cluster ID: 35df2206cd7a4422b2bb949ed972929b
I20260927 02:46:24.780879 21891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:24.786024 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:24.784807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:53379 (local address 127.0.0.1:36329)
0927 02:46:24.784846 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:24.784848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:24.785058 (+ 210us) server_negotiation.cc:407] Connection header received
0927 02:46:24.785119 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:24.785120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:24.785126 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:24.785193 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:24.785789 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:24.785790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:24.785791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:24.785795 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:24.785811 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:24.785815 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:24.785864 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:24.785926 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:46:24.785939 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:46:24.794538 21891 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:24.794744 21891 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:24.799571 21891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b382a48364d4f5fa0fcdae87d14f7b8: Generated new TSK 0
I20260927 02:46:24.799782 21891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:24.805065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 21814
2026-09-27T02:46:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:24 dist-test-slave-44kr krb5kdc[21913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:24 dist-test-slave-44kr krb5kdc[21913](info): set up 2 sockets
Sep 27 02:46:24 dist-test-slave-44kr krb5kdc[21913](info): commencing operation
krb5kdc: starting...
W20260927 02:46:25.764361 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:25 dist-test-slave-44kr krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:25.883944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46373
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:25.977243 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:25.977453 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:25.977475 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:25.978644 21926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:25.978689 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:25.978703 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:25.978713 21926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:25.978724 21926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:25.980264 21926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46373
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:25.980520 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:25.980718 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:25.985421 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:25.985451 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:25.985502 21926 server_base.cc:1034] running on GCE node
W20260927 02:46:25.985451 21934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:25.985872 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:25.986159 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:25.987309 21926 hybrid_clock.cc:648] HybridClock initialized: now 1790477185987290 us; error 37 us; skew 500 ppm
Sep 27 02:46:25 dist-test-slave-44kr krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:25.989900 21926 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:25.990391 21926 webserver.cc:460] Webserver started at http://127.0.98.62:38537/ using document root <none> and password file <none>
I20260927 02:46:25.990660 21926 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:25.990723 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:25.990850 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:25.991933 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "85e9ec083faa47758cf2ab61525a3070"
format_stamp: "Formatted at 2026-09-27 02:46:25 on dist-test-slave-44kr"
server_key: "cdd1498806c3fe828ef1d06bb1c7630b"
server_key_iv: "45c455caaea2062f0b81a519636049f6"
server_key_version: "encryptionkey@0"
I20260927 02:46:25.992264 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "85e9ec083faa47758cf2ab61525a3070"
format_stamp: "Formatted at 2026-09-27 02:46:25 on dist-test-slave-44kr"
server_key: "cdd1498806c3fe828ef1d06bb1c7630b"
server_key_iv: "45c455caaea2062f0b81a519636049f6"
server_key_version: "encryptionkey@0"
I20260927 02:46:25.993571 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:46:25.994313 21941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:25.994477 21926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:46:25.994563 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "85e9ec083faa47758cf2ab61525a3070"
format_stamp: "Formatted at 2026-09-27 02:46:25 on dist-test-slave-44kr"
server_key: "cdd1498806c3fe828ef1d06bb1c7630b"
server_key_iv: "45c455caaea2062f0b81a519636049f6"
server_key_version: "encryptionkey@0"
I20260927 02:46:25.994637 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:26.006296 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:26.008474 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:26.008628 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:26.012354 21926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46373
I20260927 02:46:26.012418 21993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46373 every 8 connection(s)
I20260927 02:46:26.012773 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:26.013574 21994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:26.015864 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070: Bootstrap starting.
I20260927 02:46:26.016453 21994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:26.016706 21994 log.cc:824] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:26.017344 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070: No bootstrap required, opened a new log
I20260927 02:46:26.018946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 21926
I20260927 02:46:26.019059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:26.018914 21994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e9ec083faa47758cf2ab61525a3070" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46373 } }
I20260927 02:46:26.019088 21994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:26.019126 21994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e9ec083faa47758cf2ab61525a3070, State: Initialized, Role: FOLLOWER
I20260927 02:46:26.019238 392 external_mini_cluster.cc:1428] Setting key e7fb63a22ce9d4a8a4dbfa419bed4921
I20260927 02:46:26.019273 21994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e9ec083faa47758cf2ab61525a3070" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46373 } }
I20260927 02:46:26.019351 21994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:26.019385 21994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:26.019424 21994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:26.020030 21994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e9ec083faa47758cf2ab61525a3070" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46373 } }
I20260927 02:46:26.020120 21994 leader_election.cc:302] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85e9ec083faa47758cf2ab61525a3070; no voters:
I20260927 02:46:26.020273 21994 leader_election.cc:288] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:26.020306 21999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:26.020452 21999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [term 1 LEADER]: Becoming Leader. State: Replica: 85e9ec083faa47758cf2ab61525a3070, State: Running, Role: LEADER
I20260927 02:46:26.020545 21994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:26.020538 21999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e9ec083faa47758cf2ab61525a3070" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46373 } }
I20260927 02:46:26.020987 22000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85e9ec083faa47758cf2ab61525a3070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e9ec083faa47758cf2ab61525a3070" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46373 } } }
I20260927 02:46:26.021114 22000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:26.021265 22001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e9ec083faa47758cf2ab61525a3070. Latest consensus state: current_term: 1 leader_uuid: "85e9ec083faa47758cf2ab61525a3070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e9ec083faa47758cf2ab61525a3070" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46373 } } }
I20260927 02:46:26.021334 22001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:26.021823 22007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:26.022261 22007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:26.023847 22007 catalog_manager.cc:1347] Generated new cluster ID: 9f8e5da9dcbd4ab5aebdf41115a5c259
I20260927 02:46:26.023900 22007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:46:26 dist-test-slave-44kr krb5kdc[21913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:26.029639 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:26.019720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52837 (local address 127.0.98.62:46373)
0927 02:46:26.020249 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:26.020260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:26.020293 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:46:26.020441 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:26.020449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:26.020507 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:26.021440 (+ 933us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:26.021560 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:26.022616 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:26.023562 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:26.024009 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:26.024111 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:26.026396 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:26.026401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:26.026401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:26.026414 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:26.028071 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:26.028469 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:26.028473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:26.028474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:26.028594 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:26.028805 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:26.028805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:26.028806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:26.029070 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:26.029242 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:26.029327 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:46:26.029432 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":388,"threads_started":1}
I20260927 02:46:26.030550 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:26.031275 22007 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:26.031993 22007 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:26.042491 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:26.032726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55649 (local address 127.0.98.62:46373)
0927 02:46:26.032765 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:26.032766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:26.036689 (+ 3923us) server_negotiation.cc:407] Connection header received
0927 02:46:26.036763 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:26.036765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:26.036776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:26.036834 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:26.037409 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:26.037926 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:26.038649 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:26.038856 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:26.038914 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:26.040120 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:26.040121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:26.040121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:26.040128 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:26.041415 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:26.041760 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:26.041761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:26.041762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:26.041809 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:26.042001 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:26.042002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:26.042002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:26.042107 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:26.042195 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:26.042337 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 02:46:26.042397 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:46:26.043396 22007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85e9ec083faa47758cf2ab61525a3070: Generated new TSK 0
I20260927 02:46:26.043566 22007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:26.046185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 21926
W20260927 02:46:26.050033 21911 connection.cc:535] client connection to 127.0.98.62:46373 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46373): Connection reset by peer (error 104)
2026-09-27T02:46:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1241 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:26 dist-test-slave-44kr krb5kdc[22025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:26 dist-test-slave-44kr krb5kdc[22025](info): set up 2 sockets
Sep 27 02:46:26 dist-test-slave-44kr krb5kdc[22025](info): commencing operation
krb5kdc: starting...
W20260927 02:46:27.012344 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:27 dist-test-slave-44kr krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:27.129437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45547
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:27.210268 22038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:27.210490 22038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:27.210515 22038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:27.211699 22038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:27.211746 22038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:27.211758 22038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:27.211769 22038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:27.211781 22038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:27.213320 22038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:27.213575 22038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:27.213788 22038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:27.218871 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:27.218855 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:27.218865 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:27.218911 22038 server_base.cc:1034] running on GCE node
I20260927 02:46:27.219304 22038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:27.219650 22038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:27.220829 22038 hybrid_clock.cc:648] HybridClock initialized: now 1790477187220812 us; error 34 us; skew 500 ppm
Sep 27 02:46:27 dist-test-slave-44kr krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:27.223523 22038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:27.223989 22038 webserver.cc:460] Webserver started at http://127.0.98.62:35745/ using document root <none> and password file <none>
I20260927 02:46:27.224223 22038 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:27.224275 22038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:27.224416 22038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:27.225512 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "30b3eb7aeca94981bab2f7d30e6b59b2"
format_stamp: "Formatted at 2026-09-27 02:46:27 on dist-test-slave-44kr"
server_key: "3c780e221979b5a88415675592ff56ef"
server_key_iv: "5c241d98bfadd597b43d62cad3b89c23"
server_key_version: "encryptionkey@0"
I20260927 02:46:27.225847 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "30b3eb7aeca94981bab2f7d30e6b59b2"
format_stamp: "Formatted at 2026-09-27 02:46:27 on dist-test-slave-44kr"
server_key: "3c780e221979b5a88415675592ff56ef"
server_key_iv: "5c241d98bfadd597b43d62cad3b89c23"
server_key_version: "encryptionkey@0"
I20260927 02:46:27.227495 22038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:27.228368 22053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:27.228591 22038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:27.228681 22038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "30b3eb7aeca94981bab2f7d30e6b59b2"
format_stamp: "Formatted at 2026-09-27 02:46:27 on dist-test-slave-44kr"
server_key: "3c780e221979b5a88415675592ff56ef"
server_key_iv: "5c241d98bfadd597b43d62cad3b89c23"
server_key_version: "encryptionkey@0"
I20260927 02:46:27.228744 22038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:27.257378 22038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:27.259379 22038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:27.259527 22038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:27.263243 22038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45547
I20260927 02:46:27.263275 22105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45547 every 8 connection(s)
I20260927 02:46:27.263602 22038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:27.264263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 22038
I20260927 02:46:27.264357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:27.264555 392 external_mini_cluster.cc:1428] Setting key 1652240833539f82ae3f4d7fb8d57cc5
I20260927 02:46:27.264636 22106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:27.268147 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2: Bootstrap starting.
I20260927 02:46:27.268857 22106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:27.269127 22106 log.cc:824] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:27.269834 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2: No bootstrap required, opened a new log
I20260927 02:46:27.270555 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:27.265101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55551 (local address 127.0.98.62:45547)
0927 02:46:27.265344 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:27.265352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:27.265380 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:46:27.266016 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:27.266025 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:27.266077 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:27.266205 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:27.266509 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:27.267481 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:27.268627 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:27.269075 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:27.269164 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:27.269828 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:27.269832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:27.269833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:27.270037 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:27.270074 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:27.270088 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:27.270179 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:27.270312 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:46:27.270340 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":108,"threads_started":1}
I20260927 02:46:27.271677 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:27.272032 22106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b3eb7aeca94981bab2f7d30e6b59b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } }
I20260927 02:46:27.272181 22106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:27.272203 22106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b3eb7aeca94981bab2f7d30e6b59b2, State: Initialized, Role: FOLLOWER
I20260927 02:46:27.272320 22106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b3eb7aeca94981bab2f7d30e6b59b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } }
I20260927 02:46:27.272410 22106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:27.272452 22106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:27.272501 22106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:27.273118 22106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b3eb7aeca94981bab2f7d30e6b59b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } }
I20260927 02:46:27.273229 22106 leader_election.cc:302] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30b3eb7aeca94981bab2f7d30e6b59b2; no voters:
I20260927 02:46:27.273422 22106 leader_election.cc:288] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:27.273456 22116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:27.273640 22116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [term 1 LEADER]: Becoming Leader. State: Replica: 30b3eb7aeca94981bab2f7d30e6b59b2, State: Running, Role: LEADER
I20260927 02:46:27.273679 22106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:27.273787 22116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b3eb7aeca94981bab2f7d30e6b59b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } }
I20260927 02:46:27.274183 22117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30b3eb7aeca94981bab2f7d30e6b59b2. Latest consensus state: current_term: 1 leader_uuid: "30b3eb7aeca94981bab2f7d30e6b59b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b3eb7aeca94981bab2f7d30e6b59b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } } }
I20260927 02:46:27.274261 22117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:27.274479 22118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30b3eb7aeca94981bab2f7d30e6b59b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b3eb7aeca94981bab2f7d30e6b59b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } } }
I20260927 02:46:27.274590 22118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:27.274905 22122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:27.275393 22122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:27.276780 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:27.273048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49641 (local address 127.0.98.62:45547)
0927 02:46:27.273089 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:27.273090 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:27.273137 (+ 47us) server_negotiation.cc:407] Connection header received
0927 02:46:27.273187 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:27.273188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:27.273198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:27.273254 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:27.273450 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:27.274136 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:27.275211 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:27.275496 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:27.275566 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:27.276457 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:27.276458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:27.276459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:27.276464 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:27.276483 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:27.276488 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:27.276551 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:27.276687 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:46:27.276700 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:46:27.277161 22122 catalog_manager.cc:1347] Generated new cluster ID: a827f887962b4fea87ae7c7773c299fa
I20260927 02:46:27.277207 22122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:27.301369 22122 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:27.302184 22122 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:27.306110 22122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30b3eb7aeca94981bab2f7d30e6b59b2: Generated new TSK 0
I20260927 02:46:27.306324 22122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:27.315958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 22038
W20260927 02:46:27.319651 22023 connection.cc:535] client connection to 127.0.98.62:45547 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45547): Connection reset by peer (error 104)
2026-09-27T02:46:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:27 dist-test-slave-44kr krb5kdc[22137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:27 dist-test-slave-44kr krb5kdc[22137](info): set up 2 sockets
Sep 27 02:46:27 dist-test-slave-44kr krb5kdc[22137](info): commencing operation
krb5kdc: starting...
W20260927 02:46:28.272298 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:28 dist-test-slave-44kr krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:28.391620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33361
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:28.470957 22150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:28.471177 22150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:28.471210 22150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:28.472349 22150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:28.472416 22150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:28.472438 22150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:28.472455 22150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:28.472479 22150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:28.473969 22150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33361
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:28.474216 22150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:28.474423 22150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:28.479190 22155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:28.479326 22150 server_base.cc:1034] running on GCE node
W20260927 02:46:28.479220 22158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:28.479434 22156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:28.479753 22150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:28.480127 22150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:28.481295 22150 hybrid_clock.cc:648] HybridClock initialized: now 1790477188481278 us; error 25 us; skew 500 ppm
Sep 27 02:46:28 dist-test-slave-44kr krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:28.483932 22150 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:28.484427 22150 webserver.cc:460] Webserver started at http://127.0.98.62:35153/ using document root <none> and password file <none>
I20260927 02:46:28.484645 22150 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:28.484707 22150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:28.484839 22150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:28.486064 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "dbcd39599a674646ad487048a3662697"
format_stamp: "Formatted at 2026-09-27 02:46:28 on dist-test-slave-44kr"
server_key: "1dae3ac025570fdd8701b1dc970ddd9a"
server_key_iv: "04a1bdcc103bc8c6106f413eb92dad0f"
server_key_version: "encryptionkey@0"
I20260927 02:46:28.486418 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "dbcd39599a674646ad487048a3662697"
format_stamp: "Formatted at 2026-09-27 02:46:28 on dist-test-slave-44kr"
server_key: "1dae3ac025570fdd8701b1dc970ddd9a"
server_key_iv: "04a1bdcc103bc8c6106f413eb92dad0f"
server_key_version: "encryptionkey@0"
I20260927 02:46:28.487882 22150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:28.488700 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:28.488889 22150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:28.488979 22150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "dbcd39599a674646ad487048a3662697"
format_stamp: "Formatted at 2026-09-27 02:46:28 on dist-test-slave-44kr"
server_key: "1dae3ac025570fdd8701b1dc970ddd9a"
server_key_iv: "04a1bdcc103bc8c6106f413eb92dad0f"
server_key_version: "encryptionkey@0"
I20260927 02:46:28.489048 22150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:28.492228 22150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:28.494149 22150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:28.494275 22150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:28.498181 22150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33361
I20260927 02:46:28.498225 22217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33361 every 8 connection(s)
I20260927 02:46:28.498533 22150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:28.499481 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:28.501718 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697: Bootstrap starting.
I20260927 02:46:28.502255 22218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:28.502504 22218 log.cc:824] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:28.503137 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697: No bootstrap required, opened a new log
I20260927 02:46:28.504706 22218 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcd39599a674646ad487048a3662697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260927 02:46:28.504843 22218 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:28.504864 22218 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcd39599a674646ad487048a3662697, State: Initialized, Role: FOLLOWER
I20260927 02:46:28.505003 22218 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcd39599a674646ad487048a3662697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260927 02:46:28.505062 22218 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:28.505084 22218 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:28.505125 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:28.505652 22218 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcd39599a674646ad487048a3662697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260927 02:46:28.505741 22218 leader_election.cc:302] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbcd39599a674646ad487048a3662697; no voters:
I20260927 02:46:28.505903 22218 leader_election.cc:288] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:28.505940 22221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:28.506129 22218 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:28.506299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 22150
I20260927 02:46:28.506322 22221 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [term 1 LEADER]: Becoming Leader. State: Replica: dbcd39599a674646ad487048a3662697, State: Running, Role: LEADER
I20260927 02:46:28.506414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:28.506413 22221 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcd39599a674646ad487048a3662697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260927 02:46:28.506592 392 external_mini_cluster.cc:1428] Setting key 378410ea0f7d25f7ad2b9bf6bd27f7b0
I20260927 02:46:28.506742 22222 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbcd39599a674646ad487048a3662697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcd39599a674646ad487048a3662697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } } }
I20260927 02:46:28.506803 22222 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:28.506942 22223 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbcd39599a674646ad487048a3662697. Latest consensus state: current_term: 1 leader_uuid: "dbcd39599a674646ad487048a3662697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcd39599a674646ad487048a3662697" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } } }
I20260927 02:46:28.506995 22223 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:28.507267 22230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:28.507845 22230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:28.509366 22230 catalog_manager.cc:1347] Generated new cluster ID: aa6f4493542e4e418b93f09c180324c9
I20260927 02:46:28.509416 22230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:28.513616 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:28.507197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52711 (local address 127.0.98.62:33361)
0927 02:46:28.507599 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:28.507608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:28.507640 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:46:28.507754 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:28.507761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:28.507792 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:28.508212 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:28.513163 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:28.513167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:28.513168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:28.513173 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:28.513228 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:28.513244 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:28.513308 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:28.513399 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 02:46:28.513439 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":55,"threads_started":1}
I20260927 02:46:28.514545 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:28.517062 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:28.516019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41761 (local address 127.0.98.62:33361)
0927 02:46:28.516058 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:28.516059 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:28.516122 (+ 63us) server_negotiation.cc:407] Connection header received
0927 02:46:28.516175 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:28.516175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:28.516180 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:28.516239 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:28.516850 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:28.516852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:28.516852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:28.516856 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:28.516873 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:28.516877 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:28.516943 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:28.516982 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 02:46:28.516995 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:46:28.521917 22230 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:28.522109 22230 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:28.533100 22230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbcd39599a674646ad487048a3662697: Generated new TSK 0
I20260927 02:46:28.533311 22230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:28.539929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 22150
2026-09-27T02:46:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:28 dist-test-slave-44kr krb5kdc[22249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:28 dist-test-slave-44kr krb5kdc[22249](info): set up 2 sockets
Sep 27 02:46:28 dist-test-slave-44kr krb5kdc[22249](info): commencing operation
krb5kdc: starting...
W20260927 02:46:29.492188 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:29 dist-test-slave-44kr krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:29.607477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46231
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:29.687206 22262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:29.687415 22262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:29.687444 22262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:29.688625 22262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:29.688668 22262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:29.688687 22262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:29.688702 22262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:29.688719 22262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:29.690189 22262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:29.690434 22262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:29.690647 22262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:29.695569 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:29.695576 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:29.695576 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:29.695945 22262 server_base.cc:1034] running on GCE node
I20260927 02:46:29.696177 22262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:29.696509 22262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:29.697657 22262 hybrid_clock.cc:648] HybridClock initialized: now 1790477189697645 us; error 31 us; skew 500 ppm
Sep 27 02:46:29 dist-test-slave-44kr krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:29.700204 22262 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:29.700706 22262 webserver.cc:460] Webserver started at http://127.0.98.62:45635/ using document root <none> and password file <none>
I20260927 02:46:29.700912 22262 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:29.700960 22262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:29.701097 22262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:29.702162 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "4ee97c3fe784465cac1ddd939508d02a"
format_stamp: "Formatted at 2026-09-27 02:46:29 on dist-test-slave-44kr"
server_key: "849b2691ca3255d72414afb9c6d35ed5"
server_key_iv: "3618d0edf61c1721dbe472b3fdc14dfd"
server_key_version: "encryptionkey@0"
I20260927 02:46:29.702497 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ee97c3fe784465cac1ddd939508d02a"
format_stamp: "Formatted at 2026-09-27 02:46:29 on dist-test-slave-44kr"
server_key: "849b2691ca3255d72414afb9c6d35ed5"
server_key_iv: "3618d0edf61c1721dbe472b3fdc14dfd"
server_key_version: "encryptionkey@0"
I20260927 02:46:29.703785 22262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:29.704591 22277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:29.704752 22262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:29.704839 22262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "4ee97c3fe784465cac1ddd939508d02a"
format_stamp: "Formatted at 2026-09-27 02:46:29 on dist-test-slave-44kr"
server_key: "849b2691ca3255d72414afb9c6d35ed5"
server_key_iv: "3618d0edf61c1721dbe472b3fdc14dfd"
server_key_version: "encryptionkey@0"
I20260927 02:46:29.704952 22262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:29.724943 22262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:29.727349 22262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:29.727540 22262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:29.731356 22262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46231
I20260927 02:46:29.731405 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46231 every 8 connection(s)
I20260927 02:46:29.731736 22262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:29.732332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 22262
I20260927 02:46:29.732503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:29.732649 22330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:29.732667 392 external_mini_cluster.cc:1428] Setting key aeb10cbbe0187ffd0e3e8593ecf974ff
I20260927 02:46:29.735980 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a: Bootstrap starting.
I20260927 02:46:29.736639 22330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:29.736922 22330 log.cc:824] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:29.737663 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a: No bootstrap required, opened a new log
Sep 27 02:46:29 dist-test-slave-44kr krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:29.739492 22330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee97c3fe784465cac1ddd939508d02a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260927 02:46:29.739638 22330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:29.739671 22330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee97c3fe784465cac1ddd939508d02a, State: Initialized, Role: FOLLOWER
I20260927 02:46:29.739801 22330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee97c3fe784465cac1ddd939508d02a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260927 02:46:29.739884 22330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:29.739908 22330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:29.739955 22330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:29.740522 22330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee97c3fe784465cac1ddd939508d02a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260927 02:46:29.740617 22330 leader_election.cc:302] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ee97c3fe784465cac1ddd939508d02a; no voters:
I20260927 02:46:29.740783 22330 leader_election.cc:288] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:29.740851 22335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:29.740962 22335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [term 1 LEADER]: Becoming Leader. State: Replica: 4ee97c3fe784465cac1ddd939508d02a, State: Running, Role: LEADER
I20260927 02:46:29.741052 22330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:29.741092 22335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee97c3fe784465cac1ddd939508d02a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260927 02:46:29.741487 22335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ee97c3fe784465cac1ddd939508d02a. Latest consensus state: current_term: 1 leader_uuid: "4ee97c3fe784465cac1ddd939508d02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee97c3fe784465cac1ddd939508d02a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } } }
I20260927 02:46:29.741515 22337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ee97c3fe784465cac1ddd939508d02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee97c3fe784465cac1ddd939508d02a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } } }
I20260927 02:46:29.741634 22335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:29.741643 22337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:29.741895 22344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:29.742478 22344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:29.743315 22333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:29.733176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53979 (local address 127.0.98.62:46231)
0927 02:46:29.733362 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:29.733372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:29.733418 (+ 46us) server_negotiation.cc:407] Connection header received
0927 02:46:29.733819 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:29.733828 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:29.733891 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:29.734041 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:29.734650 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:29.735973 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:29.736882 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:29.737414 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:29.737530 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:29.739731 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:29.739736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:29.739736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:29.739749 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:29.741547 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:29.741963 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:29.741966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:29.741966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:29.742065 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:29.742282 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:29.742282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:29.742283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:29.742596 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:29.742736 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:29.743066 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 02:46:29.743167 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260927 02:46:29.744248 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:29.744334 22344 catalog_manager.cc:1347] Generated new cluster ID: 124bca59837c4df981b3d94c71261fef
I20260927 02:46:29.744375 22344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:29.750783 22344 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:29.751498 22344 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:29.753171 22333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:29.745668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45727 (local address 127.0.98.62:46231)
0927 02:46:29.745703 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:29.745705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:29.745817 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:46:29.745906 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:29.745907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:29.745916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:29.745943 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:29.746611 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:29.747329 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:29.748350 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:29.748687 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:29.748730 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:29.750335 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:29.750336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:29.750337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:29.750343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:29.751919 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:29.752292 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:29.752293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:29.752294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:29.752348 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:29.752589 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:29.752590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:29.752590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:29.752710 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:29.752778 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:29.752977 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 02:46:29.753072 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:46:29.762236 22344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ee97c3fe784465cac1ddd939508d02a: Generated new TSK 0
I20260927 02:46:29.762444 22344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:29.764581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 22262
2026-09-27T02:46:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:29 dist-test-slave-44kr krb5kdc[22361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:29 dist-test-slave-44kr krb5kdc[22361](info): set up 2 sockets
Sep 27 02:46:29 dist-test-slave-44kr krb5kdc[22361](info): commencing operation
krb5kdc: starting...
W20260927 02:46:30.720235 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:30 dist-test-slave-44kr krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:30.835785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:30.915210 22374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:30.915412 22374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:30.915434 22374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:30.916610 22374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:30.916653 22374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:30.916666 22374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:30.916677 22374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:30.916690 22374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:30.918159 22374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:30.918383 22374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:30.918586 22374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:30.923504 22380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:30.923521 22382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:30.923575 22374 server_base.cc:1034] running on GCE node
W20260927 02:46:30.923556 22379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:30.923982 22374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:30.924286 22374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:30.925462 22374 hybrid_clock.cc:648] HybridClock initialized: now 1790477190925450 us; error 53 us; skew 500 ppm
Sep 27 02:46:30 dist-test-slave-44kr krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:30.928311 22374 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:30.928844 22374 webserver.cc:460] Webserver started at http://127.0.98.62:38911/ using document root <none> and password file <none>
I20260927 02:46:30.929082 22374 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:30.929132 22374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:30.929246 22374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:30.930305 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "5743a4b1993d4c8284b35a3221d7d3b1"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-44kr"
server_key: "3ee3c34c3a575200002eeac2eed2c46b"
server_key_iv: "42806f956a2b1aa118e0a5014f66772b"
server_key_version: "encryptionkey@0"
I20260927 02:46:30.930639 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "5743a4b1993d4c8284b35a3221d7d3b1"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-44kr"
server_key: "3ee3c34c3a575200002eeac2eed2c46b"
server_key_iv: "42806f956a2b1aa118e0a5014f66772b"
server_key_version: "encryptionkey@0"
I20260927 02:46:30.931898 22374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:30.932734 22389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:30.932911 22374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:30.933044 22374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "5743a4b1993d4c8284b35a3221d7d3b1"
format_stamp: "Formatted at 2026-09-27 02:46:30 on dist-test-slave-44kr"
server_key: "3ee3c34c3a575200002eeac2eed2c46b"
server_key_iv: "42806f956a2b1aa118e0a5014f66772b"
server_key_version: "encryptionkey@0"
I20260927 02:46:30.933111 22374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:30.949719 22374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:30.952219 22374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:30.952378 22374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:30.956251 22374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38007
I20260927 02:46:30.956344 22441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38007 every 8 connection(s)
I20260927 02:46:30.956701 22374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:30.957635 22442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:30.960017 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1: Bootstrap starting.
I20260927 02:46:30.960666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 22374
I20260927 02:46:30.960701 22442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:30.960796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:30.960979 392 external_mini_cluster.cc:1428] Setting key 14c9e966107d782a2a04c0e8c4f8ee41
I20260927 02:46:30.961032 22442 log.cc:824] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:30.961831 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1: No bootstrap required, opened a new log
I20260927 02:46:30.964496 22442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743a4b1993d4c8284b35a3221d7d3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } }
I20260927 02:46:30.964641 22442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:30.964674 22442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5743a4b1993d4c8284b35a3221d7d3b1, State: Initialized, Role: FOLLOWER
I20260927 02:46:30.964804 22442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743a4b1993d4c8284b35a3221d7d3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } }
I20260927 02:46:30.964890 22442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:30.964931 22442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:30.964982 22442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:30.965657 22442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743a4b1993d4c8284b35a3221d7d3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } }
I20260927 02:46:30.965762 22442 leader_election.cc:302] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5743a4b1993d4c8284b35a3221d7d3b1; no voters:
I20260927 02:46:30.965948 22442 leader_election.cc:288] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:30.966048 22447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:30.966217 22447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [term 1 LEADER]: Becoming Leader. State: Replica: 5743a4b1993d4c8284b35a3221d7d3b1, State: Running, Role: LEADER
I20260927 02:46:30.966259 22442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:30.966609 22447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743a4b1993d4c8284b35a3221d7d3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } }
I20260927 02:46:30.967566 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:30.961507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52097 (local address 127.0.98.62:38007)
0927 02:46:30.961697 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:30.961707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:30.961740 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:46:30.962322 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:30.962331 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:30.962407 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:30.962565 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:30.962853 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.964265 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.965254 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.965896 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.966007 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:30.966670 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:30.966675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:30.966675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:30.966875 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:30.966915 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:30.966932 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:30.966996 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:30.967292 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 02:46:30.967360 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
W20260927 02:46:30.968226 22462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 02:46:30.968348 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5743a4b1993d4c8284b35a3221d7d3b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743a4b1993d4c8284b35a3221d7d3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } }
I20260927 02:46:30.968338 22448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5743a4b1993d4c8284b35a3221d7d3b1. Latest consensus state: current_term: 1 leader_uuid: "5743a4b1993d4c8284b35a3221d7d3b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743a4b1993d4c8284b35a3221d7d3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } }
I20260927 02:46:30.968459 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:30.968461 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:30.968485 22448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1 [sys.catalog]: This master's current role is: LEADER
W20260927 02:46:30.968339 22462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:30.972827 22464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:30.973026 22464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:30.975172 22464 catalog_manager.cc:1347] Generated new cluster ID: 6527742cf07e4b5286376a95c3fe045e
I20260927 02:46:30.975229 22464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:30.976820 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:30.970006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38561 (local address 127.0.98.62:38007)
0927 02:46:30.970048 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:30.970050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:30.972690 (+ 2640us) server_negotiation.cc:407] Connection header received
0927 02:46:30.972730 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:30.972731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:30.972748 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:30.972809 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:30.972966 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.973475 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.974374 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:30.974697 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:30.974770 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:30.975536 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:30.975537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:30.975537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:30.975541 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:30.975554 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:30.975558 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:30.975600 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:30.976720 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0927 02:46:30.976746 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:46:30.998034 22464 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:30.998620 22464 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:31.000356 22464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5743a4b1993d4c8284b35a3221d7d3b1: Generated new TSK 0
I20260927 02:46:31.000572 22464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:31.003219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 22374
2026-09-27T02:46:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:31 dist-test-slave-44kr krb5kdc[22473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:31 dist-test-slave-44kr krb5kdc[22473](info): set up 2 sockets
Sep 27 02:46:31 dist-test-slave-44kr krb5kdc[22473](info): commencing operation
krb5kdc: starting...
W20260927 02:46:31.952270 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:32 dist-test-slave-44kr krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:32.065403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34911
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:32.144343 22486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:32.144579 22486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:32.144622 22486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:32.145763 22486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:32.145808 22486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:32.145825 22486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:32.145841 22486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:32.145856 22486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:32.147418 22486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:32.147689 22486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:32.147900 22486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:32.152659 22494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:32.152627 22492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:32.152762 22486 server_base.cc:1034] running on GCE node
W20260927 02:46:32.152644 22491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:32.153154 22486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:32.153402 22486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:32.154537 22486 hybrid_clock.cc:648] HybridClock initialized: now 1790477192154524 us; error 31 us; skew 500 ppm
Sep 27 02:46:32 dist-test-slave-44kr krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:32.156944 22486 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:32.157375 22486 webserver.cc:460] Webserver started at http://127.0.98.62:37149/ using document root <none> and password file <none>
I20260927 02:46:32.157564 22486 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:32.157614 22486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:32.157701 22486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:32.158672 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "d0b1c30db463485b9eb4728e45aa0497"
format_stamp: "Formatted at 2026-09-27 02:46:32 on dist-test-slave-44kr"
server_key: "e0489e305f23fcac9128df562b8f4b73"
server_key_iv: "49a0ff4069e05fadc5ca0140e8bb5009"
server_key_version: "encryptionkey@0"
I20260927 02:46:32.158954 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0b1c30db463485b9eb4728e45aa0497"
format_stamp: "Formatted at 2026-09-27 02:46:32 on dist-test-slave-44kr"
server_key: "e0489e305f23fcac9128df562b8f4b73"
server_key_iv: "49a0ff4069e05fadc5ca0140e8bb5009"
server_key_version: "encryptionkey@0"
I20260927 02:46:32.160136 22486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:46:32.160946 22501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:32.161132 22486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:46:32.161209 22486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "d0b1c30db463485b9eb4728e45aa0497"
format_stamp: "Formatted at 2026-09-27 02:46:32 on dist-test-slave-44kr"
server_key: "e0489e305f23fcac9128df562b8f4b73"
server_key_iv: "49a0ff4069e05fadc5ca0140e8bb5009"
server_key_version: "encryptionkey@0"
I20260927 02:46:32.161276 22486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:32.164371 22486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:32.166406 22486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:32.166512 22486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:32.170187 22486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34911
I20260927 02:46:32.170223 22553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34911 every 8 connection(s)
I20260927 02:46:32.170512 22486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:32.171412 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:32.173588 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497: Bootstrap starting.
I20260927 02:46:32.174096 22554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:32.174331 22554 log.cc:824] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:32.174930 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497: No bootstrap required, opened a new log
I20260927 02:46:32.176501 22554 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b1c30db463485b9eb4728e45aa0497" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34911 } }
I20260927 02:46:32.176632 22554 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:46:32.176652 22554 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b1c30db463485b9eb4728e45aa0497, State: Initialized, Role: FOLLOWER
I20260927 02:46:32.176784 22554 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b1c30db463485b9eb4728e45aa0497" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34911 } }
I20260927 02:46:32.176857 22554 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:46:32.176880 22554 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:46:32.176925 22554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:32.177421 22554 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b1c30db463485b9eb4728e45aa0497" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34911 } }
I20260927 02:46:32.177512 22554 leader_election.cc:302] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0b1c30db463485b9eb4728e45aa0497; no voters:
I20260927 02:46:32.177665 22554 leader_election.cc:288] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:46:32.177714 22557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:32.177860 22557 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [term 1 LEADER]: Becoming Leader. State: Replica: d0b1c30db463485b9eb4728e45aa0497, State: Running, Role: LEADER
I20260927 02:46:32.177891 22554 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:32.177948 22557 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b1c30db463485b9eb4728e45aa0497" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34911 } }
I20260927 02:46:32.178335 22559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0b1c30db463485b9eb4728e45aa0497. Latest consensus state: current_term: 1 leader_uuid: "d0b1c30db463485b9eb4728e45aa0497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b1c30db463485b9eb4728e45aa0497" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34911 } } }
I20260927 02:46:32.178465 22559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:32.178457 22558 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0b1c30db463485b9eb4728e45aa0497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b1c30db463485b9eb4728e45aa0497" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34911 } } }
I20260927 02:46:32.178741 22558 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:32.178850 22566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:32.179386 22566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:32.179947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 22486
I20260927 02:46:32.180066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:32.180229 392 external_mini_cluster.cc:1428] Setting key ca62b41a7509d686bb02f57c01a56159
I20260927 02:46:32.180953 22566 catalog_manager.cc:1347] Generated new cluster ID: 1f8d49ad5f5c4ff8b610753ee98236ff
I20260927 02:46:32.181006 22566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:32.182559 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.180708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50887 (local address 127.0.98.62:34911)
0927 02:46:32.181256 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:32.181265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:32.181292 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:46:32.181398 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:32.181405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:32.181435 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:32.181549 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:32.182167 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:32.182170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:32.182171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:32.182174 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:32.182206 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:32.182219 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:32.182269 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:32.182346 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:46:32.182383 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":67,"threads_started":1}
I20260927 02:46:32.185527 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:32.193507 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:32.186938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55427 (local address 127.0.98.62:34911)
0927 02:46:32.186969 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:32.186971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:32.192568 (+ 5597us) server_negotiation.cc:407] Connection header received
0927 02:46:32.192627 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:32.192628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:32.192634 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:32.192703 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:32.193292 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:32.193294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:32.193294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:46:32.193298 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:32.193314 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:46:32.193319 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:32.193375 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:32.193424 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:46:32.193437 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:46:32.195310 22566 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:32.195465 22566 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:32.197887 22566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0b1c30db463485b9eb4728e45aa0497: Generated new TSK 0
I20260927 02:46:32.198040 22566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:32.202386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 22486
2026-09-27T02:46:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1198 ms)
[----------] 18 tests from AuthTokenIssuingTest (22198 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:32 dist-test-slave-44kr krb5kdc[22585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:32 dist-test-slave-44kr krb5kdc[22585](info): set up 2 sockets
Sep 27 02:46:32 dist-test-slave-44kr krb5kdc[22585](info): commencing operation
krb5kdc: starting...
W20260927 02:46:33.148437 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:33.265509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33541
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33541,127.0.98.61:46485,127.0.98.60:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:33.345288 22598 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:46:33.345504 22598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:33.345526 22598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:33.345551 22598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:33.346689 22598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:33.346733 22598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:33.346745 22598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:33.346756 22598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:33.346766 22598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:33.348238 22598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33541,127.0.98.61:46485,127.0.98.60:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:33.348479 22598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:33.348695 22598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:33.353401 22604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:33.353585 22598 server_base.cc:1034] running on GCE node
W20260927 02:46:33.353415 22603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:33.353415 22606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:33.355741 22598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:33.356014 22598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:33.357179 22598 hybrid_clock.cc:648] HybridClock initialized: now 1790477193357154 us; error 54 us; skew 500 ppm
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:33.359541 22598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:33.359952 22598 webserver.cc:460] Webserver started at http://127.0.98.62:33067/ using document root <none> and password file <none>
I20260927 02:46:33.360188 22598 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:33.360229 22598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:33.360323 22598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:33.361362 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "2c1ded17129a4539a63734e482314f61"
format_stamp: "Formatted at 2026-09-27 02:46:33 on dist-test-slave-44kr"
server_key: "3fc4fff8c484be71440d9b232f02289b"
server_key_iv: "bb28186bef0a8c23b44c81e2b1412fd0"
server_key_version: "encryptionkey@0"
I20260927 02:46:33.361649 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c1ded17129a4539a63734e482314f61"
format_stamp: "Formatted at 2026-09-27 02:46:33 on dist-test-slave-44kr"
server_key: "3fc4fff8c484be71440d9b232f02289b"
server_key_iv: "bb28186bef0a8c23b44c81e2b1412fd0"
server_key_version: "encryptionkey@0"
I20260927 02:46:33.362833 22598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:46:33.363498 22613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:33.363677 22598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:46:33.363756 22598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "2c1ded17129a4539a63734e482314f61"
format_stamp: "Formatted at 2026-09-27 02:46:33 on dist-test-slave-44kr"
server_key: "3fc4fff8c484be71440d9b232f02289b"
server_key_iv: "bb28186bef0a8c23b44c81e2b1412fd0"
server_key_version: "encryptionkey@0"
I20260927 02:46:33.363811 22598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:33.388485 22598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:33.390772 22598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:33.390889 22598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:33.394670 22598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33541
I20260927 02:46:33.394801 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33541 every 8 connection(s)
I20260927 02:46:33.395056 22598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:33.395975 22666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:33.397572 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:33.400465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 22598
I20260927 02:46:33.400575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:33.400729 392 external_mini_cluster.cc:1428] Setting key 15eed5d2eeae945b6e27b109052802b1
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 02:46:33.407178 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.398241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33541 (local address 127.0.0.1:49364)
0927 02:46:33.398417 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.398461 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:33.398993 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:33.399256 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:33.399258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:33.399538 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:33.399688 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.399693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.400922 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.400923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.401638 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.401641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.402131 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.402132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.402195 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.402688 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:33.402697 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:33.404006 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:33.405801 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.405807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.405809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.405998 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.406248 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.406248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.406249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.406322 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.406647 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:33.406652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:33.406714 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 02:46:33.406792 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:46:33.406991 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":66,"thread_start_us":87,"threads_started":1}
I20260927 02:46:33.407181 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.398295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.0.98.62:33541)
0927 02:46:33.398448 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.398459 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.398553 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:46:33.399144 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.399146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.399167 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.399243 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.399776 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.400823 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.401727 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.402041 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.402148 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.404083 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.404090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.404090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.404104 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.405717 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.406082 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.406085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.406086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.406190 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.406377 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.406378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.406379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.406586 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.406659 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.406821 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.406933 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260927 02:46:33.407917 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:33.408450 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.408315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:38582)
0927 02:46:33.408355 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.408401 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:46:33.408578 22614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
I20260927 02:46:33.409003 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.408910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:38588)
0927 02:46:33.408955 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.408979 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:46:33.409087 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46485: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:46:33.425583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46485
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33541,127.0.98.61:46485,127.0.98.60:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:33.458328 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } attempt: 1
I20260927 02:46:33.458897 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.458718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:38600)
0927 02:46:33.458813 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.458863 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 02:46:33.459375 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.459259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:38604)
0927 02:46:33.459320 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.459350 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:46:33.459491 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46485: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
W20260927 02:46:33.505131 22673 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:46:33.505347 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:33.505362 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:33.505385 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:33.506572 22673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:33.506618 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:33.506631 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:33.506642 22673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:33.506654 22673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:33.508133 22673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33541,127.0.98.61:46485,127.0.98.60:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:33.508327 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:33.508525 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:33.513252 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:33.513257 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:33.513254 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:33.513293 22673 server_base.cc:1034] running on GCE node
I20260927 02:46:33.513700 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:33.513981 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:33.515162 22673 hybrid_clock.cc:648] HybridClock initialized: now 1790477193515148 us; error 32 us; skew 500 ppm
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:33.521001 22673 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:46:33.521502 22673 webserver.cc:460] Webserver started at http://127.0.98.61:41605/ using document root <none> and password file <none>
I20260927 02:46:33.521725 22673 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:33.521775 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:33.521900 22673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:33.522976 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/data/instance:
uuid: "fc532c2bc186461a9c29a492a69450a9"
format_stamp: "Formatted at 2026-09-27 02:46:33 on dist-test-slave-44kr"
server_key: "3e3f09e6eb80cfa936882a75f65d3b6c"
server_key_iv: "837a6ea059541e822cecc140587c8dd6"
server_key_version: "encryptionkey@0"
I20260927 02:46:33.523320 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/wal/instance:
uuid: "fc532c2bc186461a9c29a492a69450a9"
format_stamp: "Formatted at 2026-09-27 02:46:33 on dist-test-slave-44kr"
server_key: "3e3f09e6eb80cfa936882a75f65d3b6c"
server_key_iv: "837a6ea059541e822cecc140587c8dd6"
server_key_version: "encryptionkey@0"
I20260927 02:46:33.524648 22673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:46:33.525386 22688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:33.525573 22673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:33.525661 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/wal
uuid: "fc532c2bc186461a9c29a492a69450a9"
format_stamp: "Formatted at 2026-09-27 02:46:33 on dist-test-slave-44kr"
server_key: "3e3f09e6eb80cfa936882a75f65d3b6c"
server_key_iv: "837a6ea059541e822cecc140587c8dd6"
server_key_version: "encryptionkey@0"
I20260927 02:46:33.525723 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:33.527652 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } attempt: 2
I20260927 02:46:33.528031 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.527922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:38606)
0927 02:46:33.527974 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.528006 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:46:33.528482 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.528315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:38620)
0927 02:46:33.528377 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.528453 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 02:46:33.528578 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46485: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
I20260927 02:46:33.537230 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:33.539042 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:33.539182 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:33.543081 22673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46485
I20260927 02:46:33.543130 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46485 every 8 connection(s)
I20260927 02:46:33.543438 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:46:33.544365 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:33.545938 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:33.550345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 22673
I20260927 02:46:33.550455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-1/wal/instance
I20260927 02:46:33.550637 392 external_mini_cluster.cc:1428] Setting key 141523ccc1aae5831ca2005fdc771146
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:33.554626 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.546679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.0.98.62:33541)
0927 02:46:33.546760 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.546762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.546877 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:46:33.547378 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.547379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.547394 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.547446 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.548054 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.548563 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.549540 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.549743 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.549800 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.552004 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.552006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.552006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.552013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.553283 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.553734 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.553735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.553735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.553784 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.554025 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.554025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.554025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.554118 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.554160 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.554470 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.554552 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:46:33.554878 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.546572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33541 (local address 127.0.0.1:49366)
0927 02:46:33.546747 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.546794 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:33.547277 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:33.547528 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:33.547531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:33.547767 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:33.547960 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.547964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.548650 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.548651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.549448 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.549450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.549823 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.549824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.549944 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.550454 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:33.550464 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:33.551898 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:33.553358 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.553366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.553368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.553666 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.553838 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.553839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.553839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.553971 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.554172 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:33.554178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:33.554347 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 02:46:33.554464 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:46:33.554649 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260927 02:46:33.555517 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:46:33.563932 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.555777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:38632)
0927 02:46:33.555921 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.555929 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:33.555961 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:33.556105 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:33.556105 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:33.556209 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:33.556261 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.556262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.557514 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.557515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.558082 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.558085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.558573 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.558574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.558639 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.558962 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:33.558965 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:33.561110 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:33.562980 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.562981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.562981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.563139 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.563328 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.563329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.563329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.563375 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.563682 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:33.563682 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:33.563719 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:46:33.563765 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:46:33.563843 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:46:33.564059 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.555835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.0.98.61:46485)
0927 02:46:33.555962 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.555967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.555989 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:46:33.556009 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.556011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.556025 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.556104 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.556321 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.557389 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.558155 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.558480 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.558575 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.561206 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.561213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.561214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.561220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.562899 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.563210 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.563213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.563214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.563275 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.563418 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.563418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.563418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.563621 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.563697 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.563798 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.563927 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260927 02:46:33.564764 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:33.565085 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.564966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47462)
0927 02:46:33.565006 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.565056 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:46:33.565204 22689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
I20260927 02:46:33.565632 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.565544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47468)
0927 02:46:33.565586 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.565610 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:46:33.565713 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:46:33.575150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46013
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33541,127.0.98.61:46485,127.0.98.60:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:33.614959 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } attempt: 1
I20260927 02:46:33.615486 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.615331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47472)
0927 02:46:33.615399 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.615447 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 02:46:33.615854 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.615781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47476)
0927 02:46:33.615808 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.615835 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 02:46:33.615933 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
I20260927 02:46:33.619720 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } attempt: 3
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:46:33.626230 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.620048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.0.98.61:46485)
0927 02:46:33.620142 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.620144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.620153 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:46:33.620179 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.620180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.620190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.620233 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.620572 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.621048 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.621763 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.621974 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.622030 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.623823 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.623824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.623824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.623831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.625147 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.625539 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.625540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.625540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.625614 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.625798 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.625798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.625799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.625901 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.625943 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.626092 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.626164 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:46:33.626281 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.619994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:38648)
0927 02:46:33.620045 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.620057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:33.620107 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:33.620248 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:33.620248 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:33.620422 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:33.620492 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.620494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.621133 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.621134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.621687 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.621689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.622036 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.622037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.622112 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.622510 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:33.622515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:33.623734 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:33.625227 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.625228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.625229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.625449 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.625676 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.625677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.625677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.625743 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.625951 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:33.625952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:33.626004 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 02:46:33.626075 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:46:33.626182 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:46:33.626767 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:33.627087 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.626985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47482)
0927 02:46:33.627030 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.627063 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:46:33.627406 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.627320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47494)
0927 02:46:33.627360 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.627385 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:46:33.627492 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
W20260927 02:46:33.656332 22748 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:46:33.656558 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:33.656581 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:33.656605 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:33.657733 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:33.657774 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:33.657791 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:33.657807 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:33.657820 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:33.658627 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } attempt: 1
I20260927 02:46:33.659019 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.658906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47508)
0927 02:46:33.658958 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.658990 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:46:33.659368 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.659281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47516)
0927 02:46:33.659323 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.659350 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:46:33.659411 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33541,127.0.98.61:46485,127.0.98.60:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
W20260927 02:46:33.659461 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
I20260927 02:46:33.659662 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:33.659854 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:33.664569 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:33.664572 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:33.664563 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:33.664835 22748 server_base.cc:1034] running on GCE node
I20260927 02:46:33.665082 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:33.665347 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:33.666495 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790477193666483 us; error 27 us; skew 500 ppm
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:33.668962 22748 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:46:33.669402 22748 webserver.cc:460] Webserver started at http://127.0.98.60:43551/ using document root <none> and password file <none>
I20260927 02:46:33.669588 22748 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:33.669629 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:33.669719 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:33.670710 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/data/instance:
uuid: "ad96fbd3648a4f21935e50c0a317bfc4"
format_stamp: "Formatted at 2026-09-27 02:46:33 on dist-test-slave-44kr"
server_key: "d767e11168065e5131035c53cfb61c7f"
server_key_iv: "d18599236ac862bfccda2c662fb02868"
server_key_version: "encryptionkey@0"
I20260927 02:46:33.670993 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/wal/instance:
uuid: "ad96fbd3648a4f21935e50c0a317bfc4"
format_stamp: "Formatted at 2026-09-27 02:46:33 on dist-test-slave-44kr"
server_key: "d767e11168065e5131035c53cfb61c7f"
server_key_iv: "d18599236ac862bfccda2c662fb02868"
server_key_version: "encryptionkey@0"
I20260927 02:46:33.672420 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:46:33.673076 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:33.673218 22748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:46:33.673298 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/wal
uuid: "ad96fbd3648a4f21935e50c0a317bfc4"
format_stamp: "Formatted at 2026-09-27 02:46:33 on dist-test-slave-44kr"
server_key: "d767e11168065e5131035c53cfb61c7f"
server_key_iv: "d18599236ac862bfccda2c662fb02868"
server_key_version: "encryptionkey@0"
I20260927 02:46:33.673352 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:33.684068 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } attempt: 2
I20260927 02:46:33.684569 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.684433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47522)
0927 02:46:33.684495 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.684536 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:46:33.684996 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.684871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47524)
0927 02:46:33.684942 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.684972 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 02:46:33.685086 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46013: connect: Connection refused (error 111)
I20260927 02:46:33.694720 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:33.696905 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:33.697027 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:33.700929 22748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46013
I20260927 02:46:33.700976 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46013 every 8 connection(s)
I20260927 02:46:33.701308 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:46:33.702265 22816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:33.703781 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:33.709985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 22748
I20260927 02:46:33.710079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476929252373-392-0/minicluster-data/master-2/wal/instance
I20260927 02:46:33.710233 392 external_mini_cluster.cc:1428] Setting key fd4dcb3b422c747b1b297679e59c3655
I20260927 02:46:33.713172 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.704455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33541 (local address 127.0.0.1:49370)
0927 02:46:33.704617 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.704659 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:33.705101 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:33.705297 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:33.705300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:33.705538 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:33.705728 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.705732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.706382 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.706384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.707321 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.707323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.707670 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.707671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.707783 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.708351 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:33.708359 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:33.709841 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:33.711732 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.711736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.711738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.712090 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.712278 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.712279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.712279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.712423 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.712676 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:33.712680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:33.712813 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 02:46:33.712896 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:46:33.713013 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260927 02:46:33.713198 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.704499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.0.98.62:33541)
0927 02:46:33.704549 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.704551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.704737 (+ 186us) server_negotiation.cc:407] Connection header received
0927 02:46:33.705192 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.705193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.705202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.705238 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.705787 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.706286 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.707405 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.707591 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.707638 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.710000 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.710002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.710002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.710009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.711661 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.712169 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.712170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.712170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.712224 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.712490 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.712491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.712491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.712612 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.712673 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.713024 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.713103 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:46:33.713744 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:46:33.719216 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.710571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.0.98.60:46013)
0927 02:46:33.710724 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.710727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.710787 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:46:33.710860 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.710863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.710877 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.710953 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.711650 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.712659 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.713596 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.713982 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.714044 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.716627 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.716635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.716636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.716643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.718073 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.718411 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.718414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.718414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.718471 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.718631 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.718631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.718632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.718794 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.718921 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.719038 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.719111 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260927 02:46:33.720216 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.713966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:38656)
0927 02:46:33.714004 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.714014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:33.714049 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:33.714189 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:33.714189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:33.714291 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:33.714333 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.714334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.715157 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.715158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.715742 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.715744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.716197 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.716199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.716276 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.716666 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:33.716670 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:33.717700 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:33.719189 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.719190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.719190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.719380 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.719594 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.719595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.719595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.719662 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.719975 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:33.719976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:33.720019 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:46:33.720088 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:46:33.720157 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 02:46:33.720281 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.714041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.0.98.61:46485)
0927 02:46:33.714089 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.714091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.714098 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:46:33.714121 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.714122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.714131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.714186 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.714400 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.715069 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.715838 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.716036 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.716143 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.717776 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.717778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.717779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.717786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.719111 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.719473 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.719475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.719475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.719537 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.719727 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.719728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.719728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.719865 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.719919 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.720120 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.720198 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:46:33.720628 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:33.728291 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.719993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.0.98.62:33541)
0927 02:46:33.720036 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.720038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.720088 (+ 50us) server_negotiation.cc:407] Connection header received
0927 02:46:33.720231 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.720232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.720242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.720270 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.720939 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.721612 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.722417 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.722753 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.722817 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.725859 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.725860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.725861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.725868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.727223 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.727527 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.727529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.727529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.727580 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.727758 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.727760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.727760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.727914 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.728086 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.728121 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.728196 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:46:33.728808 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.720975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47538)
0927 02:46:33.721039 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.721049 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:33.721092 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:33.721248 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:33.721249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:33.721380 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:33.721415 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.721416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.722166 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.722166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.722825 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.722827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.723256 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.723256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.723298 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.723628 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:33.723632 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:33.725383 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:33.727649 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.727650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.727650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.727827 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.727997 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.727998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.727998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.728045 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.728413 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:33.728414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:33.728453 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 02:46:33.728507 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:46:33.728714 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:46:33.728870 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.720999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.98.60:46013)
0927 02:46:33.721029 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.721031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.721081 (+ 50us) server_negotiation.cc:407] Connection header received
0927 02:46:33.721166 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.721167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.721175 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.721220 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.721474 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.722062 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.722900 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.723168 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.723218 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.725476 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.725477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.725478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.725485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.727458 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.727882 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.727883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.727884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.727930 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.728099 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.728100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.728100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.728250 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.728374 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.728616 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.728777 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":15}
I20260927 02:46:33.730644 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4: Bootstrap starting.
I20260927 02:46:33.731590 22816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:33.731894 22816 log.cc:824] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:33.732761 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4: No bootstrap required, opened a new log
I20260927 02:46:33.734578 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } attempt: 2
I20260927 02:46:33.734835 22816 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } }
I20260927 02:46:33.734984 22816 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad96fbd3648a4f21935e50c0a317bfc4, State: Initialized, Role: FOLLOWER
I20260927 02:46:33.735060 22816 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } }
I20260927 02:46:33.735376 22816 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:33.735791 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:33.735868 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:46:33.737308 22833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:33.737360 22833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:46:33.737937 22748 master_runner.cc:186] Error getting master registration for 127.0.98.62:33541: OK, instance_id { permanent_uuid: "2c1ded17129a4539a63734e482314f61" instance_seqno: 1790477193393578 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:46:33.740474 22748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:46:33.743151 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.735007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.0.98.61:46485)
0927 02:46:33.735048 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.735050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.735060 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:46:33.735084 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.735085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.735094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.735149 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.735739 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.736498 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.737575 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.737995 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.738074 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.740681 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.740683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.740683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.740690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.742156 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.742454 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.742455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.742456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.742512 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.742671 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.742672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.742672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.742797 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.742962 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.743026 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.743089 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:46:33.744325 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:33.745211 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.734847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47548)
0927 02:46:33.735092 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.735102 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:33.735138 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:33.737122 (+ 1984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:33.737124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:33.737282 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:33.737323 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.737325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.738145 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.738146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.738855 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.738858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.739334 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.739335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.739406 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.740725 (+ 1319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:33.740730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:33.741975 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:33.744088 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.744089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.744090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.744294 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.744562 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.744562 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.744563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.744628 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.744920 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:33.744921 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:33.744968 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 02:46:33.745044 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:46:33.745138 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260927 02:46:33.745239 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.735218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.98.60:46013)
0927 02:46:33.736960 (+ 1742us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.736962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.736984 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:46:33.737022 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.737023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.737032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.737380 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.737408 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.738063 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.738951 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.739241 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.739302 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.742051 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.742053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.742053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.742060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.743990 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.744373 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.744374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.744374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.744492 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.744692 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.744693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.744693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.744846 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.744914 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.745080 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.745154 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1722}
I20260927 02:46:33.745765 22640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2c1ded17129a4539a63734e482314f61"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49372
I20260927 02:46:33.747643 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61: Bootstrap starting.
I20260927 02:46:33.748332 22666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:33.748641 22666 log.cc:824] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:33.749461 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61: No bootstrap required, opened a new log
I20260927 02:46:33.750800 22666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } }
I20260927 02:46:33.750936 22666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c1ded17129a4539a63734e482314f61, State: Initialized, Role: FOLLOWER
I20260927 02:46:33.751008 22666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } }
I20260927 02:46:33.751247 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:33.751317 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:33.751318 22666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:33.751448 22640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:46:33.751529 22640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:33.752266 22640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } }
I20260927 02:46:33.753034 22790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c1ded17129a4539a63734e482314f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ad96fbd3648a4f21935e50c0a317bfc4"
I20260927 02:46:33.753175 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [term 0 FOLLOWER]: Advancing to term 1
W20260927 02:46:33.753357 22847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:33.753409 22847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:33.754067 22790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c1ded17129a4539a63734e482314f61 in term 1.
I20260927 02:46:33.754374 22614 leader_election.cc:302] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c1ded17129a4539a63734e482314f61, ad96fbd3648a4f21935e50c0a317bfc4; no voters:
I20260927 02:46:33.754526 22836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:33.754695 22836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [term 1 LEADER]: Becoming Leader. State: Replica: 2c1ded17129a4539a63734e482314f61, State: Running, Role: LEADER
I20260927 02:46:33.752535 22640 leader_election.cc:288] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [CANDIDATE]: Term 1 election: Requested vote from peers fc532c2bc186461a9c29a492a69450a9 (127.0.98.61:46485), ad96fbd3648a4f21935e50c0a317bfc4 (127.0.98.60:46013)
I20260927 02:46:33.754782 22836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } }
I20260927 02:46:33.755319 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c1ded17129a4539a63734e482314f61. Latest consensus state: current_term: 1 leader_uuid: "2c1ded17129a4539a63734e482314f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:33.755391 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:33.756949 22853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:33.757129 22853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:33.754760 22598 master_runner.cc:186] Error getting master registration for 127.0.98.61:46485: OK, instance_id { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" instance_seqno: 1790477193542003 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:46:33.757560 22598 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:46:33.758971 22790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [term 1 FOLLOWER]: Refusing update from remote peer 2c1ded17129a4539a63734e482314f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:33.759459 22849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:33.760854 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c1ded17129a4539a63734e482314f61. Latest consensus state: current_term: 1 leader_uuid: "2c1ded17129a4539a63734e482314f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:33.760943 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:33.764446 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.755929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.0.98.62:33541)
0927 02:46:33.755976 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.755978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.756570 (+ 592us) server_negotiation.cc:407] Connection header received
0927 02:46:33.756709 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.756710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.756720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.756750 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.757990 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.758656 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.759652 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.759911 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.759951 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.761492 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.761493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.761493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.761500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.763035 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.763425 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.763426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.763427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.763480 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.763680 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.763681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.763681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.763793 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.763893 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.764003 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.764078 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:46:33.768972 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.756157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.0.98.61:46485)
0927 02:46:33.756201 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.756203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.757518 (+ 1315us) server_negotiation.cc:407] Connection header received
0927 02:46:33.758050 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.758051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.758062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.758090 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.760533 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.761245 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.762248 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.762564 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.762632 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.765825 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.765827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.765827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.765834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.767368 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.767762 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.767763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.767764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.767822 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.768030 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.768031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.768031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.768170 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.768740 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.768799 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.768877 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:46:33.776201 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } attempt: 3
I20260927 02:46:33.776916 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.756276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.0.98.60:46013)
0927 02:46:33.756328 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.756330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.759371 (+ 3041us) server_negotiation.cc:407] Connection header received
0927 02:46:33.759754 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.759755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.759765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.759793 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.762280 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.762983 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.764299 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.764600 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.764662 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.773783 (+ 9121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.773784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.773785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.773792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.775314 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.775653 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.775654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.775654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.775706 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.775891 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.775892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.775892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.776033 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.776126 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.776746 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.776830 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:46:33.777114 22849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c1ded17129a4539a63734e482314f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:33.777253 22849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:33.777818 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c1ded17129a4539a63734e482314f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:33.777907 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:33.777971 22859 mvcc.cc:205] Tried to move back new op lower bound from 7333794585634050048 to 7333794585620451328. Current Snapshot: MvccSnapshot[applied={T|T < 7333794585634050048}]
I20260927 02:46:33.780843 22853 catalog_manager.cc:1347] Generated new cluster ID: f0b35725e532445c8eae1608a3a6cee5
I20260927 02:46:33.780900 22853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:46:33 dist-test-slave-44kr krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:46:33.784291 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.776654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46013 (local address 127.0.0.1:47574)
0927 02:46:33.776688 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:33.776710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:33.776749 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:33.777174 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:33.777176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:33.777328 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:33.777375 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.777377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.778168 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.778169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.778995 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:33.778998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.779440 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.779441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:33.779503 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.779887 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:33.779892 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:33.781259 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:33.783281 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.783282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.783283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.783494 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.783667 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:33.783668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:33.783669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:33.783733 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:33.783981 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:33.783982 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:33.784025 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:46:33.784085 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:46:33.784194 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:46:33.784301 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:33.776505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.0.98.60:46013)
0927 02:46:33.777009 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.777011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.777019 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:33.777047 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.777048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.777057 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.777101 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.777448 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.778107 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.779090 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:33.779383 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:33.779438 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:33.781377 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:33.781378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:33.781379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:33.781386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:33.783219 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.783543 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.783556 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.783557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.783607 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:33.783780 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:33.783780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:33.783781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:33.783914 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:33.783976 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:33.784163 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 02:46:33.784222 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":103,"threads_started":1}
I20260927 02:46:33.785827 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9: Bootstrap starting.
I20260927 02:46:33.786259 22853 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:33.786554 22741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:33.786851 22741 log.cc:824] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:33.786952 22853 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:33.787949 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9: No bootstrap required, opened a new log
I20260927 02:46:33.789861 22741 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } }
I20260927 02:46:33.789994 22741 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc532c2bc186461a9c29a492a69450a9, State: Initialized, Role: FOLLOWER
I20260927 02:46:33.790083 22741 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } }
I20260927 02:46:33.790362 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:33.790576 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:33.790647 22741 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:33.790720 22715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c1ded17129a4539a63734e482314f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fc532c2bc186461a9c29a492a69450a9"
I20260927 02:46:33.790817 22715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:33.791673 22715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c1ded17129a4539a63734e482314f61 in term 1.
I20260927 02:46:33.791869 22714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [term 1 FOLLOWER]: Refusing update from remote peer 2c1ded17129a4539a63734e482314f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 02:46:33.792809 22878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:33.792856 22878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:33.793288 22836 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:33.795418 22853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61: Generated new TSK 0
I20260927 02:46:33.795648 22853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:46:33.795845 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c1ded17129a4539a63734e482314f61. Latest consensus state: current_term: 1 leader_uuid: "2c1ded17129a4539a63734e482314f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:33.795989 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:33.796066 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c1ded17129a4539a63734e482314f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:33.796159 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:33.797008 22881 mvcc.cc:205] Tried to move back new op lower bound from 7333794585765511168 to 7333794585620451328. Current Snapshot: MvccSnapshot[applied={T|T < 7333794585634050048}]
W20260927 02:46:33.818364 22669 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.816762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.98.62:33541)
0927 02:46:33.816820 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.816822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.817710 (+ 888us) server_negotiation.cc:407] Connection header received
0927 02:46:33.817810 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.817811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.817822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.817850 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.818289 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49394: BlockingRecv error: recv got EOF from 127.0.0.1:49394 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:46:33.819099 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.817176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.0.98.61:46485)
0927 02:46:33.817216 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.817217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.818608 (+ 1391us) server_negotiation.cc:407] Connection header received
0927 02:46:33.818697 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.818697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.818707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.818734 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.819021 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38680: BlockingRecv error: recv got EOF from 127.0.0.1:38680 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 02:46:33.819766 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.817476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.0.98.60:46013)
0927 02:46:33.817533 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.817534 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.819320 (+ 1786us) server_negotiation.cc:407] Connection header received
0927 02:46:33.819401 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.819403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.819412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.819438 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.819708 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47588: BlockingRecv error: recv got EOF from 127.0.0.1:47588 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 02:46:33.825435 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.824663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.0.98.61:46485)
0927 02:46:33.824699 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.824700 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.824929 (+ 229us) server_negotiation.cc:407] Connection header received
0927 02:46:33.825037 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.825038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.825057 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.825083 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.825387 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38692: BlockingRecv error: recv got EOF from 127.0.0.1:38692 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:46:33.828557 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.827720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.0.98.61:46485)
0927 02:46:33.827754 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.827755 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.827968 (+ 213us) server_negotiation.cc:407] Connection header received
0927 02:46:33.828064 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.828065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.828075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.828104 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.828510 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38696: BlockingRecv error: recv got EOF from 127.0.0.1:38696 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:46:33.832355 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.831688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.0.98.61:46485)
0927 02:46:33.831719 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.831721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.831906 (+ 185us) server_negotiation.cc:407] Connection header received
0927 02:46:33.831991 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.831992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.832001 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.832027 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.832310 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38698: BlockingRecv error: recv got EOF from 127.0.0.1:38698 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260927 02:46:33.838346 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.837576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.0.98.61:46485)
0927 02:46:33.837611 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.837613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.837832 (+ 219us) server_negotiation.cc:407] Connection header received
0927 02:46:33.837926 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.837927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.837938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.837967 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.838294 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38704: BlockingRecv error: recv got EOF from 127.0.0.1:38704 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:46:33.849864 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.848864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.98.61:46485)
0927 02:46:33.848908 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.848909 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.849228 (+ 319us) server_negotiation.cc:407] Connection header received
0927 02:46:33.849333 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.849334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.849347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.849377 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.849806 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38706: BlockingRecv error: recv got EOF from 127.0.0.1:38706 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 02:46:33.868194 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.867288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.0.98.61:46485)
0927 02:46:33.867335 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.867338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.867608 (+ 270us) server_negotiation.cc:407] Connection header received
0927 02:46:33.867708 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.867709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.867722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.867752 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.868136 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38716: BlockingRecv error: recv got EOF from 127.0.0.1:38716 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 02:46:33.902231 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.901555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.0.98.61:46485)
0927 02:46:33.901598 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.901600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.901717 (+ 117us) server_negotiation.cc:407] Connection header received
0927 02:46:33.901771 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.901771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.901783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.901828 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.902165 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38718: BlockingRecv error: recv got EOF from 127.0.0.1:38718 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 02:46:33.968019 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:33.967314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.0.98.61:46485)
0927 02:46:33.967378 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:33.967380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:33.967464 (+ 84us) server_negotiation.cc:407] Connection header received
0927 02:46:33.967515 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:33.967516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:33.967527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:33.967631 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:33.967961 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38722: BlockingRecv error: recv got EOF from 127.0.0.1:38722 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 02:46:34.097729 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:34.097131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.0.98.61:46485)
0927 02:46:34.097197 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:34.097199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:34.097271 (+ 72us) server_negotiation.cc:407] Connection header received
0927 02:46:34.097326 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:34.097326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:34.097337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:34.097386 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:34.097669 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38726: BlockingRecv error: recv got EOF from 127.0.0.1:38726 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:46:34.322696 22941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c1ded17129a4539a63734e482314f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1ded17129a4539a63734e482314f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } peers { permanent_uuid: "fc532c2bc186461a9c29a492a69450a9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "ad96fbd3648a4f21935e50c0a317bfc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46013 } } }
I20260927 02:46:34.322834 22941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c1ded17129a4539a63734e482314f61 [sys.catalog]: This master's current role is: LEADER
W20260927 02:46:34.355767 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:34.355131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.0.98.61:46485)
0927 02:46:34.355190 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:34.355192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:34.355202 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:46:34.355226 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:34.355227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:34.355237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:34.355287 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:34.355695 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38734: BlockingRecv error: recv got EOF from 127.0.0.1:38734 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:46:34.737788 22833 catalog_manager.cc:1259] Loaded cluster ID: f0b35725e532445c8eae1608a3a6cee5
I20260927 02:46:34.737866 22833 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4: loading cluster ID for follower catalog manager: success
I20260927 02:46:34.738699 22833 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4: acquiring CA information for follower catalog manager: success
I20260927 02:46:34.738996 22833 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad96fbd3648a4f21935e50c0a317bfc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:46:34.793268 22878 catalog_manager.cc:1259] Loaded cluster ID: f0b35725e532445c8eae1608a3a6cee5
I20260927 02:46:34.793346 22878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9: loading cluster ID for follower catalog manager: success
I20260927 02:46:34.794185 22878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9: acquiring CA information for follower catalog manager: success
I20260927 02:46:34.794486 22878 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc532c2bc186461a9c29a492a69450a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:46:34.871565 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:34.868949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.0.98.61:46485)
0927 02:46:34.869078 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:34.869080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:34.869111 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:46:34.869200 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:34.869200 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:34.869264 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:34.869329 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:34.869521 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.870033 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.870805 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.871005 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.871059 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:34.871220 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:34.871333 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:34.871403 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:34.871444 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 02:46:34.871467 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260927 02:46:34.875916 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:34.873517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.0.98.60:46013)
0927 02:46:34.873625 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:34.873627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:34.873635 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:34.873672 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:34.873673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:34.873707 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:34.873774 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:34.873882 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.874353 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.875123 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.875320 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.875374 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:34.875545 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:34.875693 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:34.875777 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:34.875801 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 02:46:34.875826 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260927 02:46:34.880446 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:34.877625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.0.98.61:46485)
0927 02:46:34.877677 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:34.877679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:34.877900 (+ 221us) server_negotiation.cc:407] Connection header received
0927 02:46:34.877952 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:34.877953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:34.877965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:34.878006 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:34.878142 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.878667 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.879720 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.880004 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.880052 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:34.880192 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:34.880229 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:34.880271 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:34.880323 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 02:46:34.880345 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:46:34.880501 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:34.877765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.0.98.60:46013)
0927 02:46:34.877806 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:34.877808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:34.877997 (+ 189us) server_negotiation.cc:407] Connection header received
0927 02:46:34.878086 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:34.878087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:34.878131 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:34.878182 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:34.878266 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.878913 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.879606 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.879881 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.879945 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:34.880149 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:34.880181 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:34.880223 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:34.880407 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 02:46:34.880431 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:46:34.881111 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:34.877545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.0.98.62:33541)
0927 02:46:34.877632 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:34.877634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:34.877643 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:46:34.877756 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:34.877757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:34.877804 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:34.877877 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:34.877993 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.878768 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.880084 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:34.880496 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:34.880549 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:34.880771 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:34.880871 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:34.880923 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:34.881024 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:46:34.881050 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":44,"threads_started":1}
I20260927 02:46:34.882575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 22598
I20260927 02:46:34.887054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 22673
I20260927 02:46:34.891083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 22748
2026-09-27T02:46:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2689 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:46:34 dist-test-slave-44kr krb5kdc[22972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:46:34 dist-test-slave-44kr krb5kdc[22972](info): set up 2 sockets
Sep 27 02:46:34 dist-test-slave-44kr krb5kdc[22972](info): commencing operation
krb5kdc: starting...
W20260927 02:46:35.844447 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/test-user.keytab.
Sep 27 02:46:35 dist-test-slave-44kr krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:46:35.967075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43541
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43541,127.0.98.61:43209,127.0.98.60:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:46:36.047122 22985 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:46:36.047343 22985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:36.047386 22985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:36.047431 22985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:36.048669 22985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:36.048715 22985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:36.048728 22985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:36.048739 22985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:36.048750 22985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:36.050266 22985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43541,127.0.98.61:43209,127.0.98.60:41199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:36.050506 22985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:36.050736 22985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:36.055684 22993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:36.055708 22990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:36.055706 22991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:36.055943 22985 server_base.cc:1034] running on GCE node
I20260927 02:46:36.056246 22985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:36.056558 22985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:36.057723 22985 hybrid_clock.cc:648] HybridClock initialized: now 1790477196057704 us; error 38 us; skew 500 ppm
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:36.060541 22985 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:46:36.061035 22985 webserver.cc:460] Webserver started at http://127.0.98.62:37875/ using document root <none> and password file <none>
I20260927 02:46:36.061235 22985 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:36.061278 22985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:36.061386 22985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:36.062420 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/data/instance:
uuid: "09d3b81edf274fce88f13702e502da27"
format_stamp: "Formatted at 2026-09-27 02:46:36 on dist-test-slave-44kr"
server_key: "6ce2ae2c16397c1d18e56a528d63cdab"
server_key_iv: "8e217c6d31fecc6178bb9803b21ac9ab"
server_key_version: "encryptionkey@0"
I20260927 02:46:36.062749 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/wal/instance:
uuid: "09d3b81edf274fce88f13702e502da27"
format_stamp: "Formatted at 2026-09-27 02:46:36 on dist-test-slave-44kr"
server_key: "6ce2ae2c16397c1d18e56a528d63cdab"
server_key_iv: "8e217c6d31fecc6178bb9803b21ac9ab"
server_key_version: "encryptionkey@0"
I20260927 02:46:36.064037 22985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:46:36.064807 23000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:36.064993 22985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:46:36.065075 22985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/wal
uuid: "09d3b81edf274fce88f13702e502da27"
format_stamp: "Formatted at 2026-09-27 02:46:36 on dist-test-slave-44kr"
server_key: "6ce2ae2c16397c1d18e56a528d63cdab"
server_key_iv: "8e217c6d31fecc6178bb9803b21ac9ab"
server_key_version: "encryptionkey@0"
I20260927 02:46:36.065147 22985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:36.076922 22985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:36.078897 22985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:36.079039 22985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:36.082929 22985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43541
I20260927 02:46:36.083096 23052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43541 every 8 connection(s)
I20260927 02:46:36.083312 22985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:46:36.084347 23053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:36.085981 23053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:36.092262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 22985
I20260927 02:46:36.092355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-0/wal/instance
I20260927 02:46:36.092598 392 external_mini_cluster.cc:1428] Setting key 46c884063c13563732cf4078a749e781
I20260927 02:46:36.096329 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.086707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.0.98.62:43541)
0927 02:46:36.086858 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.086864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.086913 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:46:36.087471 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.087474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.087503 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.087574 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.088098 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.088965 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.089792 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.090116 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.090209 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.092500 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.092509 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.092509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.092517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.094474 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.094971 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.094974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.094975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.095064 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.095326 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.095327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.095328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.095608 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.095682 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.095932 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.096117 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260927 02:46:36.096338 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.086628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43541 (local address 127.0.0.1:35036)
0927 02:46:36.086773 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.086818 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:36.087323 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:36.087588 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:36.087591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:36.087846 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:36.088012 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.088016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.089047 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.089048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.089713 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.089714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.090209 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.090210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.090274 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.090875 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:36.090883 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:36.092354 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:36.094584 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.094593 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.094595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.094886 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.095128 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.095129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.095130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.095241 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.095679 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:36.095685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:36.095767 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 02:46:36.095868 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 02:46:36.096185 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":156,"thread_start_us":75,"threads_started":1}
I20260927 02:46:36.097417 23053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:36.097798 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.097674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43209 (local address 127.0.0.1:41582)
0927 02:46:36.097722 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.097772 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 02:46:36.097939 23001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
I20260927 02:46:36.098361 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.098261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43209 (local address 127.0.0.1:41598)
0927 02:46:36.098307 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.098339 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:46:36.098469 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43209: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:46:36.119632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43209
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43541,127.0.98.61:43209,127.0.98.60:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:36.147706 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } attempt: 1
I20260927 02:46:36.148275 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.148091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43209 (local address 127.0.0.1:41608)
0927 02:46:36.148160 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.148232 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 02:46:36.148662 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.148570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43209 (local address 127.0.0.1:41622)
0927 02:46:36.148609 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.148638 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:46:36.148818 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43209: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
W20260927 02:46:36.200529 23060 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:46:36.200759 23060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:36.200778 23060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:36.200799 23060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:36.202003 23060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:36.202059 23060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:36.202075 23060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:36.202087 23060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:36.202100 23060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:36.203634 23060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43541,127.0.98.61:43209,127.0.98.60:41199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:36.203878 23060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:36.204116 23060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:36.208900 23068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:36.208942 23066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:36.208988 23065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:36.209097 23060 server_base.cc:1034] running on GCE node
I20260927 02:46:36.209326 23060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:36.209620 23060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:36.210770 23060 hybrid_clock.cc:648] HybridClock initialized: now 1790477196210756 us; error 25 us; skew 500 ppm
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:36.213317 23060 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:46:36.213753 23060 webserver.cc:460] Webserver started at http://127.0.98.61:33371/ using document root <none> and password file <none>
I20260927 02:46:36.213994 23060 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:36.214043 23060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:36.214161 23060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:36.215201 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/data/instance:
uuid: "5eca8f67165e418fa6c398a59e568df3"
format_stamp: "Formatted at 2026-09-27 02:46:36 on dist-test-slave-44kr"
server_key: "de3f96d34f3432352c62390601356585"
server_key_iv: "8383d53b0300cdafa73db6c70c842284"
server_key_version: "encryptionkey@0"
I20260927 02:46:36.215523 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/wal/instance:
uuid: "5eca8f67165e418fa6c398a59e568df3"
format_stamp: "Formatted at 2026-09-27 02:46:36 on dist-test-slave-44kr"
server_key: "de3f96d34f3432352c62390601356585"
server_key_iv: "8383d53b0300cdafa73db6c70c842284"
server_key_version: "encryptionkey@0"
I20260927 02:46:36.216933 23060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 02:46:36.216997 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } attempt: 2
I20260927 02:46:36.217340 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.217231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43209 (local address 127.0.0.1:41624)
0927 02:46:36.217280 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.217310 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:46:36.217701 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.217606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43209 (local address 127.0.0.1:41636)
0927 02:46:36.217649 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.217676 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:46:36.217753 23075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260927 02:46:36.217809 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43209: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43209: connect: Connection refused (error 111)
I20260927 02:46:36.217944 23060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:46:36.218022 23060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/wal
uuid: "5eca8f67165e418fa6c398a59e568df3"
format_stamp: "Formatted at 2026-09-27 02:46:36 on dist-test-slave-44kr"
server_key: "de3f96d34f3432352c62390601356585"
server_key_iv: "8383d53b0300cdafa73db6c70c842284"
server_key_version: "encryptionkey@0"
I20260927 02:46:36.218091 23060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:36.247632 23060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:36.249987 23060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:36.250135 23060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:36.253854 23060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43209
I20260927 02:46:36.253893 23127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43209 every 8 connection(s)
I20260927 02:46:36.254211 23060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:46:36.254455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 23060
I20260927 02:46:36.254554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-1/wal/instance
I20260927 02:46:36.254786 392 external_mini_cluster.cc:1428] Setting key f415bcf9651e181f0648132c2b1f4faf
I20260927 02:46:36.255349 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:36.257172 23128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:46:36.268285 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.257865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.0.98.62:43541)
0927 02:46:36.257929 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.257931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.258103 (+ 172us) server_negotiation.cc:407] Connection header received
0927 02:46:36.258587 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.258589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.258603 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.258654 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.259251 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.259735 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.260796 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.261001 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.261067 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.265051 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.265053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.265053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.265062 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.266591 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.267082 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.267083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.267083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.267143 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.267448 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.267449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.267449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.267605 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.267681 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.268142 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.268217 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:46:36.268467 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.257819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43541 (local address 127.0.0.1:35044)
0927 02:46:36.257975 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.258018 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:36.258491 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:36.258722 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:36.258726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:36.258971 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:36.259164 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.259168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.259817 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.259818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.260717 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.260720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.261082 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.261083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.261203 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.261787 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:36.261796 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:36.264947 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:36.266688 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.266697 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.266699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.267000 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.267214 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.267215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.267216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.267356 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.267675 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:36.267682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:36.267889 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 02:46:36.268008 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 02:46:36.268192 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260927 02:46:36.269057 23128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:46:36.277290 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.269301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43209 (local address 127.0.0.1:41646)
0927 02:46:36.269341 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.269352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:36.269387 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:36.269675 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:36.269677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:36.269835 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:36.269906 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.269907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.270789 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.270791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.271590 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.271593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.272079 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.272081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.272160 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.272706 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:36.272711 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:36.274550 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:36.276154 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.276155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.276155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.276341 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.276642 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.276643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.276643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.276699 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.276989 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:36.276990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:36.277034 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 02:46:36.277144 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 02:46:36.277220 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 02:46:36.277407 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.269359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.0.98.61:43209)
0927 02:46:36.269539 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.269543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.269560 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:46:36.269579 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.269581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.269593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.269648 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.270003 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.270707 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.271665 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.271971 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.272026 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.274640 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.274648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.274648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.274654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.276068 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.276513 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.276516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.276516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.276578 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.276749 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.276749 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.276750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.276903 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.276969 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.277217 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.277281 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":126,"threads_started":1}
I20260927 02:46:36.277956 23128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:36.278266 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.278161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39152)
0927 02:46:36.278203 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.278248 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 02:46:36.278390 23076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:46:36.278800 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.278721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39164)
0927 02:46:36.278758 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.278778 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:46:36.278887 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
I20260927 02:46:36.279316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKApS4J/build/release/bin/kudu
/tmp/dist-test-taskKApS4J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41199
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43541,127.0.98.61:43209,127.0.98.60:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:46:36.309001 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } attempt: 3
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:46:36.316310 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.309461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.98.61:43209)
0927 02:46:36.309518 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.309521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.309536 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:46:36.309625 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.309626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.309636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.309685 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.310059 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.310598 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.311543 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.311792 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.311855 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.313870 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.313872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.313872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.313881 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.315266 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.315594 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.315595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.315595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.315649 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.315811 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.315812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.315813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.315930 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.315994 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.316128 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.316217 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:46:36.316310 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.309369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43209 (local address 127.0.0.1:41650)
0927 02:46:36.309476 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.309490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:36.309563 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:36.309705 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:36.309706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:36.309890 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:36.309962 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.309964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.310674 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.310675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.311463 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.311466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.311865 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.311866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.311946 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.312370 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:36.312376 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:36.313783 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:36.315353 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.315354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.315354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.315528 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.315701 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.315702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.315702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.315761 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.315984 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:36.315985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:36.316037 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 02:46:36.316099 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:46:36.316205 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 02:46:36.316803 23053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:36.317132 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.317041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39174)
0927 02:46:36.317083 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.317113 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:46:36.317421 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.317340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39190)
0927 02:46:36.317377 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.317400 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:46:36.317502 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
I20260927 02:46:36.328176 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } attempt: 1
I20260927 02:46:36.328691 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.328546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39206)
0927 02:46:36.328606 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.328652 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:46:36.329037 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.328954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39210)
0927 02:46:36.328991 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.329017 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:46:36.329157 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
I20260927 02:46:36.348675 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } attempt: 1
I20260927 02:46:36.349169 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.349023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39214)
0927 02:46:36.349092 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.349137 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 02:46:36.349583 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.349455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39230)
0927 02:46:36.349507 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.349550 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 02:46:36.349740 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41199: connect: Connection refused (error 111)
W20260927 02:46:36.360735 23135 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:46:36.360937 23135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:46:36.360952 23135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:46:36.360970 23135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:46:36.362082 23135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:46:36.362119 23135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:46:36.362131 23135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:46:36.362142 23135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:46:36.362154 23135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:46:36.363590 23135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43541,127.0.98.61:43209,127.0.98.60:41199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:46:36.363806 23135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:46:36.364023 23135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:46:36.368652 23141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:46:36.368726 23140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:36.368772 23135 server_base.cc:1034] running on GCE node
W20260927 02:46:36.368667 23143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:46:36.369099 23135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:46:36.369371 23135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:46:36.370525 23135 hybrid_clock.cc:648] HybridClock initialized: now 1790477196370503 us; error 39 us; skew 500 ppm
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:46:36.372936 23135 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:46:36.373376 23135 webserver.cc:460] Webserver started at http://127.0.98.60:33173/ using document root <none> and password file <none>
I20260927 02:46:36.373580 23135 fs_manager.cc:359] Metadata directory not provided
I20260927 02:46:36.373622 23135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:46:36.373714 23135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:46:36.374696 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/data/instance:
uuid: "c8dcf3a5d16645a794384c2145af908f"
format_stamp: "Formatted at 2026-09-27 02:46:36 on dist-test-slave-44kr"
server_key: "beeee9fa0bd77bfbf4c4356d9dbd9852"
server_key_iv: "87f6adc4b1432454a986233e97ee8a41"
server_key_version: "encryptionkey@0"
I20260927 02:46:36.374989 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/wal/instance:
uuid: "c8dcf3a5d16645a794384c2145af908f"
format_stamp: "Formatted at 2026-09-27 02:46:36 on dist-test-slave-44kr"
server_key: "beeee9fa0bd77bfbf4c4356d9dbd9852"
server_key_iv: "87f6adc4b1432454a986233e97ee8a41"
server_key_version: "encryptionkey@0"
I20260927 02:46:36.376142 23135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:46:36.376875 23150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:36.377013 23135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:46:36.377096 23135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/wal
uuid: "c8dcf3a5d16645a794384c2145af908f"
format_stamp: "Formatted at 2026-09-27 02:46:36 on dist-test-slave-44kr"
server_key: "beeee9fa0bd77bfbf4c4356d9dbd9852"
server_key_iv: "87f6adc4b1432454a986233e97ee8a41"
server_key_version: "encryptionkey@0"
I20260927 02:46:36.377151 23135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:46:36.388497 23135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:46:36.390530 23135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:46:36.390650 23135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:46:36.394270 23135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41199
I20260927 02:46:36.394304 23202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41199 every 8 connection(s)
I20260927 02:46:36.394641 23135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:46:36.395570 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:46:36.397089 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:36.397315 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } attempt: 2
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:46:36.403816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKApS4J/build/release/bin/kudu as pid 23135
I20260927 02:46:36.403920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKApS4J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476929252373-392-0/minicluster-data/master-2/wal/instance
I20260927 02:46:36.404126 392 external_mini_cluster.cc:1428] Setting key 94c4c3d021fd51d1deee1f47b797b278
I20260927 02:46:36.405839 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.397789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.0.98.62:43541)
0927 02:46:36.397831 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.397833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.398055 (+ 222us) server_negotiation.cc:407] Connection header received
0927 02:46:36.398554 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.398556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.398568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.398641 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.399184 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.399664 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.400571 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.400762 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.400810 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.402722 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.402724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.402725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.402732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.404300 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.404774 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.404776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.404776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.404839 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.405143 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.405144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.405144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.405287 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.405373 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.405675 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.405751 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:46:36.405972 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.397604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39232)
0927 02:46:36.397671 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.397681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:36.397727 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:36.398565 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:36.398567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:36.398730 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:36.398770 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.398772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.399954 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.399955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.400512 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.400515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.400997 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.400998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.401043 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.401519 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:36.401524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:36.402999 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:36.404968 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.404969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.404970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.405163 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.405329 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.405329 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.405330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.405389 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.405679 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:36.405680 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:36.405723 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:46:36.405782 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:46:36.405869 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:46:36.405988 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.397748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43541 (local address 127.0.0.1:35056)
0927 02:46:36.397916 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.397966 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:36.398425 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:36.398662 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:36.398664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:36.398951 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:36.399079 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.399083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.399737 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.399738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.400498 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.400501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.400842 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.400843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.400951 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.401374 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:36.401382 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:36.402634 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:36.404372 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.404406 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.404408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.404702 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.404907 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.404907 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.404908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.405045 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.405361 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:36.405365 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:36.405529 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 02:46:36.405630 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 02:46:36.405776 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"spinlock_wait_cycles":8448,"thread_start_us":90,"threads_started":1}
I20260927 02:46:36.406250 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.397748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.0.98.60:41199)
0927 02:46:36.397925 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.397934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.397963 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:46:36.398410 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.398419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.398444 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.398534 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.398841 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.399807 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.400587 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.400925 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.400984 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.403065 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.403073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.403074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.403080 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.404882 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.405225 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.405227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.405228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.405276 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.405451 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.405451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.405452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.405589 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.405652 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.406099 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.406167 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260927 02:46:36.406797 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:46:36.408139 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3: Bootstrap starting.
I20260927 02:46:36.408888 23128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:36.409202 23128 log.cc:824] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:36.409971 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3: No bootstrap required, opened a new log
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:46:36.411907 23128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } }
I20260927 02:46:36.412050 23128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eca8f67165e418fa6c398a59e568df3, State: Initialized, Role: FOLLOWER
I20260927 02:46:36.412144 23128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } }
I20260927 02:46:36.412489 23128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:36.412969 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.413072 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:46:36.414419 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.404522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.0.98.60:41199)
0927 02:46:36.404993 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.404994 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.404998 (+ 4us) server_negotiation.cc:407] Connection header received
0927 02:46:36.405010 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.405011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.405020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.405067 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.405756 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.406645 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.407627 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.407824 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.407880 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.410466 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.410467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.410468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.410474 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.412502 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.412801 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.412802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.412803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.412855 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.413618 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.413619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.413619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.413745 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.413878 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.414252 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.414324 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":76,"threads_started":1}
W20260927 02:46:36.415048 23221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:36.415096 23221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:36.416364 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.407011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43209 (local address 127.0.0.1:41660)
0927 02:46:36.407044 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.407052 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:36.407081 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:36.409646 (+ 2565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:36.409647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:36.409783 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:36.409828 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.409830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.410702 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.410703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.411469 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.411471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.411944 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.411945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.412006 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.412376 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:36.412380 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:36.413709 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:36.415497 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.415498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.415498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.415647 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.415807 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.415807 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.415808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.415860 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.416090 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:36.416091 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:36.416130 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 02:46:36.416190 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:46:36.416272 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 02:46:36.416368 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.409469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.0.98.61:43209)
0927 02:46:36.409524 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.409526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.409534 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:36.409554 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.409554 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.409563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.409627 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.409963 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.410612 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.411559 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.411832 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.411895 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.413775 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.413776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.413777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.413783 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.415436 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.415705 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.415705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.415706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.415756 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.415914 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.415915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.415915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.416034 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.416088 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.416228 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.416289 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:46:36.414708 23060 master_runner.cc:186] Error getting master registration for 127.0.98.62:43541: OK, instance_id { permanent_uuid: "09d3b81edf274fce88f13702e502da27" instance_seqno: 1790477196081818 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:46:36.417203 23060 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:46:36.417506 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:46:36.423425 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.415216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.0.98.62:43541)
0927 02:46:36.415253 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.415254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.415261 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:46:36.415282 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.415283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.415293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.415344 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.415907 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.416650 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.417367 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.417628 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.417685 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.420241 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.420243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.420244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.420251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.421962 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.422282 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.422283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.422283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.422345 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.422828 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.422829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.422829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.422974 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.423147 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.423207 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.423281 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:46:36.424891 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.417710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39250)
0927 02:46:36.417776 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.417784 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:36.417811 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:36.418092 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:36.418092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:36.418189 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:36.418215 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.418217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.418781 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.418782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.419258 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.419259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.419710 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.419711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.419766 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.420160 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:36.420165 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:36.421607 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:36.423870 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.423871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.423871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.424069 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.424274 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.424275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.424275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.424328 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.424590 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:36.424591 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:36.424628 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:46:36.424691 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:46:36.424792 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 02:46:36.424896 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } attempt: 2
I20260927 02:46:36.425709 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.417954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.0.98.60:41199)
0927 02:46:36.418001 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.418002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.418009 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:46:36.418026 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.418026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.418035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.418078 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.418279 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.418721 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.419320 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.419537 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.419665 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.421717 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.421718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.421718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.421725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.423461 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.424149 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.424150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.424150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.424211 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.424407 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.424408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.424409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.424519 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.424593 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.424728 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.425621 (+ 893us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":799,"server-negotiator.queue_time_us":28}
I20260927 02:46:36.427263 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f: Bootstrap starting.
I20260927 02:46:36.428015 23203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:36.428341 23203 log.cc:824] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:46:36 dist-test-slave-44kr krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:46:36.431201 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f: No bootstrap required, opened a new log
I20260927 02:46:36.431960 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.424066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.0.98.61:43209)
0927 02:46:36.424121 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.424122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.424132 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:46:36.424189 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.424191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.424201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.424261 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.424819 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.425475 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.426516 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.426768 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.426860 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.429467 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.429469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.429469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.429477 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.430871 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.431235 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.431236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.431236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.431300 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.431485 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.431486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.431486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.431599 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.431742 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.431785 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.431874 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:46:36.433301 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:46:36.433704 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.425100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41199 (local address 127.0.0.1:39266)
0927 02:46:36.425144 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:46:36.425152 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:46:36.425183 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:46:36.425447 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:46:36.425448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:46:36.425594 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:46:36.425631 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.425632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.426751 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.426752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.427772 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:46:36.427774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.428222 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.428223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:46:36.428275 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.428703 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:46:36.428707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:46:36.430016 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:46:36.432734 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.432735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.432735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.432936 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.433115 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:46:36.433116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:46:36.433116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:46:36.433167 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:46:36.433401 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:46:36.433402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:46:36.433455 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 02:46:36.433519 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:46:36.433606 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:46:36.433740 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.425280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.0.98.60:41199)
0927 02:46:36.425329 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.425331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.425339 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:46:36.425358 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.425359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.425369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.425426 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.425717 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.426659 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.427864 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.428143 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.430093 (+ 1950us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.430134 (+ 41us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.430135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.430135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.430140 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.432650 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.433007 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.433008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.433008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.433058 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.433221 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.433222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.433222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.433328 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.433378 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.433575 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.433655 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:46:36.435254 23053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27: Bootstrap starting.
I20260927 02:46:36.435202 23203 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } }
I20260927 02:46:36.434430 23027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "09d3b81edf274fce88f13702e502da27"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35062
I20260927 02:46:36.435345 23203 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8dcf3a5d16645a794384c2145af908f, State: Initialized, Role: FOLLOWER
I20260927 02:46:36.435420 23203 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } }
I20260927 02:46:36.435746 23203 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:36.435976 23053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27: Neither blocks nor log segments found. Creating new log.
I20260927 02:46:36.436286 23053 log.cc:824] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27: Log is configured to *not* fsync() on all Append() calls
I20260927 02:46:36.437147 23053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27: No bootstrap required, opened a new log
W20260927 02:46:36.437721 23236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:36.437770 23236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:36.438369 23053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } }
I20260927 02:46:36.438491 23053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d3b81edf274fce88f13702e502da27, State: Initialized, Role: FOLLOWER
I20260927 02:46:36.438570 23053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } }
I20260927 02:46:36.438796 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.438833 23053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:46:36.438908 23027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:46:36.439015 23027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:36.439988 23027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } }
I20260927 02:46:36.440219 23027 leader_election.cc:288] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [CANDIDATE]: Term 1 election: Requested vote from peers 5eca8f67165e418fa6c398a59e568df3 (127.0.98.61:43209), c8dcf3a5d16645a794384c2145af908f (127.0.98.60:41199)
I20260927 02:46:36.440690 23102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d3b81edf274fce88f13702e502da27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5eca8f67165e418fa6c398a59e568df3"
I20260927 02:46:36.440826 23102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:36.441406 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:36.441659 23102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d3b81edf274fce88f13702e502da27 in term 1.
I20260927 02:46:36.441879 23177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d3b81edf274fce88f13702e502da27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c8dcf3a5d16645a794384c2145af908f"
I20260927 02:46:36.441974 23177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:46:36.442041 23001 leader_election.cc:302] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d3b81edf274fce88f13702e502da27, 5eca8f67165e418fa6c398a59e568df3; no voters:
I20260927 02:46:36.439366 23135 master_runner.cc:186] Error getting master registration for 127.0.98.62:43541: OK, instance_id { permanent_uuid: "09d3b81edf274fce88f13702e502da27" instance_seqno: 1790477196081818 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:46:36.442390 23135 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:46:36.442739 23177 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d3b81edf274fce88f13702e502da27 in term 1.
I20260927 02:46:36.443001 23238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:46:36.443164 23238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [term 1 LEADER]: Becoming Leader. State: Replica: 09d3b81edf274fce88f13702e502da27, State: Running, Role: LEADER
I20260927 02:46:36.443266 23238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } }
I20260927 02:46:36.443571 22985 master_runner.cc:263] No leader master found from master 09d3b81edf274fce88f13702e502da27
I20260927 02:46:36.443624 22985 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:46:36.443737 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d3b81edf274fce88f13702e502da27. Latest consensus state: current_term: 1 leader_uuid: "09d3b81edf274fce88f13702e502da27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.443820 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [sys.catalog]: This master's current role is: LEADER
W20260927 02:46:36.444093 23249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:46:36.444147 23249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:46:36.444823 23256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:46:36.444980 23256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:46:36.446372 23177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [term 1 FOLLOWER]: Refusing update from remote peer 09d3b81edf274fce88f13702e502da27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:36.446735 23238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:36.448450 23102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [term 1 FOLLOWER]: Refusing update from remote peer 09d3b81edf274fce88f13702e502da27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:46:36.448467 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09d3b81edf274fce88f13702e502da27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.448562 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:36.448606 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d3b81edf274fce88f13702e502da27. Latest consensus state: current_term: 1 leader_uuid: "09d3b81edf274fce88f13702e502da27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.448652 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:36.448892 23250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:46:36.450244 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d3b81edf274fce88f13702e502da27. Latest consensus state: current_term: 1 leader_uuid: "09d3b81edf274fce88f13702e502da27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.450367 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:36.453737 23258 mvcc.cc:205] Tried to move back new op lower bound from 7333794596642291712 to 7333794596632530944. Current Snapshot: MvccSnapshot[applied={T|T < 7333794596642291712}]
I20260927 02:46:36.454039 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09d3b81edf274fce88f13702e502da27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.454128 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:36.456503 23259 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09d3b81edf274fce88f13702e502da27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.456638 23259 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:36.456707 23260 mvcc.cc:205] Tried to move back new op lower bound from 7333794596642291712 to 7333794596632530944. Current Snapshot: MvccSnapshot[applied={T|T < 7333794596642291712}]
I20260927 02:46:36.457948 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09d3b81edf274fce88f13702e502da27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.458036 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27 [sys.catalog]: This master's current role is: LEADER
I20260927 02:46:36.458644 23262 mvcc.cc:205] Tried to move back new op lower bound from 7333794596642291712 to 7333794596632530944. Current Snapshot: MvccSnapshot[applied={T|T < 7333794596642291712}]
I20260927 02:46:36.459095 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09d3b81edf274fce88f13702e502da27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d3b81edf274fce88f13702e502da27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43541 } } peers { permanent_uuid: "5eca8f67165e418fa6c398a59e568df3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43209 } } peers { permanent_uuid: "c8dcf3a5d16645a794384c2145af908f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41199 } } }
I20260927 02:46:36.459187 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:46:36.464872 23256 catalog_manager.cc:1347] Generated new cluster ID: ea1e909a648f43e485bdc5e0c49644a6
I20260927 02:46:36.464928 23256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:46:36.473444 23256 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:46:36.474107 23256 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:46:36.482648 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.456643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.0.98.61:43209)
0927 02:46:36.456688 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.456690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.457173 (+ 483us) server_negotiation.cc:407] Connection header received
0927 02:46:36.457302 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.457304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.457316 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.457349 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.458278 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.459012 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.462837 (+ 3825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.463129 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.463175 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.466510 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.466511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.466512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.466519 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.468293 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.480891 (+ 12598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.480893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.480893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.480976 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.481532 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.481534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.481534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.481678 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.481784 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.482455 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.482547 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:46:36.482971 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.456988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.0.98.60:41199)
0927 02:46:36.457023 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.457025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.471452 (+ 14427us) server_negotiation.cc:407] Connection header received
0927 02:46:36.471544 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.471545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.471556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.471583 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.472165 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.472923 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.473894 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.474185 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.474222 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.480142 (+ 5920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.480144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.480145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.480152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.481813 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.482190 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.482191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.482191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.482245 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.482428 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.482429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.482429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.482558 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.482623 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.482810 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.482882 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:46:36.484697 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:36.456810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.0.98.62:43541)
0927 02:46:36.456851 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.456853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.459271 (+ 2418us) server_negotiation.cc:407] Connection header received
0927 02:46:36.459297 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.459298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.459310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.459383 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.461134 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.461834 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.463589 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:36.463849 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:36.463893 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:36.468593 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:46:36.468594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:46:36.468594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:46:36.468602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:46:36.470256 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.481224 (+ 10968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.481226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.481226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.481290 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:46:36.481980 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:46:36.481981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:46:36.481981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:46:36.482124 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:46:36.482230 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:36.484494 (+ 2264us) server_negotiation.cc:299] Negotiation successful
0927 02:46:36.484601 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:46:36.487699 23256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09d3b81edf274fce88f13702e502da27: Generated new TSK 0
I20260927 02:46:36.487934 23256 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:46:36.507501 23056 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.505748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.0.98.62:43541)
0927 02:46:36.505796 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.505798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.506770 (+ 972us) server_negotiation.cc:407] Connection header received
0927 02:46:36.506891 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.506893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.506905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.506939 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.507425 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35082: BlockingRecv error: recv got EOF from 127.0.0.1:35082 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 02:46:36.508592 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.506164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.0.98.61:43209)
0927 02:46:36.506204 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.506206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.508008 (+ 1802us) server_negotiation.cc:407] Connection header received
0927 02:46:36.508116 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.508117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.508129 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.508161 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.508526 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41704: BlockingRecv error: recv got EOF from 127.0.0.1:41704 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 02:46:36.509480 23205 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.506493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.0.98.60:41199)
0927 02:46:36.506536 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.506538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.508947 (+ 2409us) server_negotiation.cc:407] Connection header received
0927 02:46:36.509050 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.509051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.509063 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.509094 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.509416 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39284: BlockingRecv error: recv got EOF from 127.0.0.1:39284 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 02:46:36.511950 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.511153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.0.98.61:43209)
0927 02:46:36.511190 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.511191 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.511423 (+ 232us) server_negotiation.cc:407] Connection header received
0927 02:46:36.511537 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.511539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.511557 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.511589 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.511900 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41716: BlockingRecv error: recv got EOF from 127.0.0.1:41716 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:46:36.515118 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.514345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.0.98.61:43209)
0927 02:46:36.514390 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.514392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.514619 (+ 227us) server_negotiation.cc:407] Connection header received
0927 02:46:36.514722 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.514723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.514733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.514763 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.515070 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41722: BlockingRecv error: recv got EOF from 127.0.0.1:41722 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 02:46:36.519125 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.518337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.0.98.61:43209)
0927 02:46:36.518387 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.518388 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.518615 (+ 227us) server_negotiation.cc:407] Connection header received
0927 02:46:36.518719 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.518720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.518732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.518762 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.519079 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41738: BlockingRecv error: recv got EOF from 127.0.0.1:41738 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 02:46:36.525296 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.524477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.0.98.61:43209)
0927 02:46:36.524517 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.524519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.524746 (+ 227us) server_negotiation.cc:407] Connection header received
0927 02:46:36.524848 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.524849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.524861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.524891 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.525243 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41754: BlockingRecv error: recv got EOF from 127.0.0.1:41754 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 02:46:36.535777 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.534818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.0.98.61:43209)
0927 02:46:36.534868 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.534870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.535134 (+ 264us) server_negotiation.cc:407] Connection header received
0927 02:46:36.535246 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.535247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.535260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.535293 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.535714 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41764: BlockingRecv error: recv got EOF from 127.0.0.1:41764 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 02:46:36.554502 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.553493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.0.98.61:43209)
0927 02:46:36.553540 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.553542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.553834 (+ 292us) server_negotiation.cc:407] Connection header received
0927 02:46:36.553952 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.553954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.553967 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.554000 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.554438 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41778: BlockingRecv error: recv got EOF from 127.0.0.1:41778 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 02:46:36.589182 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.588174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.0.98.61:43209)
0927 02:46:36.588224 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.588226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.588545 (+ 319us) server_negotiation.cc:407] Connection header received
0927 02:46:36.588662 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.588663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.588675 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.588707 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.589115 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41790: BlockingRecv error: recv got EOF from 127.0.0.1:41790 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 02:46:36.655683 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.654883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.0.98.61:43209)
0927 02:46:36.654955 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.654957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.655083 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:46:36.655121 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.655122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.655135 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.655194 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.655620 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41804: BlockingRecv error: recv got EOF from 127.0.0.1:41804 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:46:36.785718 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:36.784974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.0.98.61:43209)
0927 02:46:36.785076 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:36.785079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:36.785157 (+ 78us) server_negotiation.cc:407] Connection header received
0927 02:46:36.785230 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:36.785231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:36.785243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:36.785294 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:36.785651 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41806: BlockingRecv error: recv got EOF from 127.0.0.1:41806 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 02:46:37.043452 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:46:37.042847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.0.98.61:43209)
0927 02:46:37.042913 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:37.042916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:37.042926 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:46:37.042971 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:37.042972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:37.042983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:37.043028 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:46:37.043388 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41822: BlockingRecv error: recv got EOF from 127.0.0.1:41822 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:46:37.415531 23221 catalog_manager.cc:1259] Loaded cluster ID: ea1e909a648f43e485bdc5e0c49644a6
I20260927 02:46:37.415624 23221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3: loading cluster ID for follower catalog manager: success
I20260927 02:46:37.416400 23221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3: acquiring CA information for follower catalog manager: success
I20260927 02:46:37.416664 23221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5eca8f67165e418fa6c398a59e568df3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:46:37.438241 23236 catalog_manager.cc:1259] Loaded cluster ID: ea1e909a648f43e485bdc5e0c49644a6
I20260927 02:46:37.438314 23236 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f: loading cluster ID for follower catalog manager: success
I20260927 02:46:37.439287 23236 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f: acquiring CA information for follower catalog manager: success
I20260927 02:46:37.439591 23236 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8dcf3a5d16645a794384c2145af908f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:46:37.559190 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:37.556685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.0.98.61:43209)
0927 02:46:37.556819 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:37.556821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:37.556832 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:46:37.556890 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:37.556891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:37.556981 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:37.557055 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:37.557194 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.557702 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.558457 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.558671 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.558737 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:37.558848 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:37.559006 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:37.559067 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:37.559086 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 02:46:37.559104 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260927 02:46:37.563331 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:37.561034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.0.98.60:41199)
0927 02:46:37.561152 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:37.561154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:37.561161 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:46:37.561202 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:37.561202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:37.561238 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:37.561307 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:37.561402 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.561896 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.562596 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.562811 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.562863 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:37.563001 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:37.563116 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:37.563179 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:37.563227 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 02:46:37.563247 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":49,"threads_started":1}
I20260927 02:46:37.567523 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:37.564762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.0.98.61:43209)
0927 02:46:37.564799 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:37.564801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:37.564980 (+ 179us) server_negotiation.cc:407] Connection header received
0927 02:46:37.565053 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:37.565054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:37.565067 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:37.565119 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:37.565243 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.565969 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.566819 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.567060 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.567127 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:37.567264 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:37.567307 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:37.567350 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:37.567424 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:46:37.567446 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:46:37.567955 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:37.564962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.0.98.60:41199)
0927 02:46:37.564998 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:37.565000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:37.565111 (+ 111us) server_negotiation.cc:407] Connection header received
0927 02:46:37.565209 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:37.565210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:37.565220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:37.565266 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:37.565475 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.566150 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.567205 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.567461 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.567512 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:37.567686 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:37.567716 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:37.567744 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:37.567864 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 02:46:37.567888 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:46:37.568137 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:46:37.564818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.0.98.62:43541)
0927 02:46:37.564916 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:46:37.564918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:46:37.565116 (+ 198us) server_negotiation.cc:407] Connection header received
0927 02:46:37.565183 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:46:37.565184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:46:37.565235 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:46:37.565311 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:46:37.565425 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.565947 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.566992 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:46:37.567274 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:46:37.567330 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:46:37.567770 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:46:37.567879 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:46:37.567954 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:46:37.568025 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:46:37.568049 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":47,"threads_started":1}
I20260927 02:46:37.569578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 22985
I20260927 02:46:37.574143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 23060
I20260927 02:46:37.578281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKApS4J/build/release/bin/kudu with pid 23135
2026-09-27T02:46:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2687 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5377 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (268330 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST